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ABSTRACT 

One  of  the  main  reasons  computer  networks  are  a major 
area  of  great  attention  and  development  today  is  their  ca- 
pability to  provide  the  facilities  for  common  use  of  data 
bases  and  information  files  by  all  computers  in  the  system. 

When  a file  is  used  by  several  computers  in  the  network, 
it  can  be  stored  in  the  memory  of  at  least  one  of  them  and 
can  be  accessed  by  the  other  computers  via  the  communication 
channels.  In  reneral  the  cost  or  querying  is  reduced  as 
we  increase  the  number  of  copies  in  the  system.  On  the 
other  hand,  storage  costs,  limitations  on  the  size  of  the 
memories  and  the  cost  of  updating  (every  copy  must  be  updated) 
will  dictate  decreasing  of  the  number  of  copies.  Further- 
more if  the  parameters  of  the  system  are  time-varying,  or  if 
the  exact  pattern  of  the  rates  of  demand  is  unknown  or  some 
non  negligible  possibility  of  node  or  link  failures  is 
expected,  then  some  kind  of  dynamic  approach  must  be  used. 

This  thesis  considers  the  problem  of  optimal  dynamic 
file  allocation  when  more  than  one  copy  is  allowed  to  exist 
in  the  system  at  any  given  time.  A general  model  to  handle 
this  problem  including  updating  traffic  and  the  possibility 
of  node  failures  will  be  developed.  The  evolution  of  the 
system  is  represented  as  a finite  state  Markov  process  and 
Dynamic  programming  will  be  used  for  the  solution  of  the 
optimization  problem. 

The  use  of  two  types  of  control  variables,  one  for 
adding  new  copies  to  the  system  and  the  other  for  erasing 
copies,  gives  the  model  certain  properties  that  permit  the 
construction  of  an  efficient  algorithm  to  solve  the  optimi- 
zation problem.  Within  the  framework  of  the  developed 
model  the  addition  of  the  updating  traffic  and  the  possibi- 
lity of  node  failures  present  no  important  difficulties. 
Furthermore  the  model  can  easily  handle  the  problem  of  cons- 
traints in  the  maximum  or  minimum  number  of  copies.  In  the 
last  chapter  the  model  and  algorithms  are  applied  to  several 
numerical  examples. 
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CHAPTER  I 


INTRODUCTION 


1-1  General  Setting  and  Description  of  the  Problem 


"The  time  sharing  industry  dominated  the  sixties 
and  it  appears  that  computer  networks  will  play  a similar 
role  in  the  seventies.  The  need  has  now  arisen  for  many 
of  these  time-shared  systems  to  share  each  others'  resources 
by  coupling  them  together  over  a communication  network 
thereby  creating  a computer  network"  (L.  Kleinrock [l]  ) . 

We  define  a computer  network  to  be  an  interconnected 
group  of  independent  computer  systems  communicating  with 
each  other  and  sharing  resources  such  as  programs,  data, 
hardware,  and  software. 

The  increasing  interest  in  this  area  is  the  cause  for 
a continuously  growing  number  of  articles,  books  and  projects 
related  to  computer  networks  [2]  - [7] , [24] . The  reasons 
why  these  types  of  networks  are  attractive  are  widely 
exposed  throughout  the  literature  in  this  field. 

a)  sharing  of  data  base,  hardware  resources,  program 
and  load 

b)  remote  data  processing 

c)  access  to  specialized  resources 

d)  recovery  of  information  from  a remote  node  in  case 


of  node  failure 


e)  decentralization  of  operations  and  need  to  trans- 
fer information  from  one  point  to  another  etc. 

One  of  the  main  reasons  computer  networks  are  a major 
area  of  great  attention  and  development  today  is  their 
capability  to  provide  the  facilities  for  common  use  of  data 
bases  and  information  files  by  all  computers  in  the  system. 
This  work  deals  with  the  problem  of  the  information  alloca- 
tion to  be  shared  by  the  computers  in  the  network.  Such 
a network  is  displayed  in  Fig.  1 

When  a file  is  used  by  several  computers  in  the  network 
it  can  be  stored  in  the  memory  of  (at  least)  one  of  them 
and  can  be  accessed  by  the  other  computers  via  the  communi- 
cation channels.  In  general,  the  cost  of  querying  is 
reduced  as  we  increase  the  number  of  copies  in  the  system. 

On  the  ether  hand,  storage  costs,  limitations  on  the  size 
of  the  memories  and  the  cost  of  updating  (every  copy 
must  be  updated)  will  dictate  decreasing  of  the  number  of 
copies . 

The  problem  of  how  many  copies  of  the  files  are  to  be 
kept  and  their  allocation  is  the  main  subject  of  this  Thesis 
Most  of  the  previous  work  in  the  area  of  file  alloca- 
tion has  been  devoted  to  the  analysis  of  the  problem  under 
static  approximations,  that  is,  assuming  that  all  parameters 
of  the  system  are  known  a priori  and  basing  the  design  on 
their  average  value  over  the  period  of  operation  of  the 
system.  The  location  of  the  files  is  then  considered  fixed 


tWl/ 


* 
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How  many  copies  of  each  file  do 
we  need  in  the  network? 

At  which  computer  do  we  have  to 
allocate  each  copy? 

For  how  long  must  a certain  alio 
cation  distribution  remain 
unchangeable? 

etc. 


Fig. 1.1.  General  representation  of  a computer  network 
reflecting  the  problem  of  file  allocation 
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for  the  whole  operating  period. 

An  early  work  in  this  field  was  a paper  by  Chu  [8]  . 

The  criterion  of  optimality  used  in  [8]  , is  minimal  overall 
operating  costs.  The  model  considers  storage  and  transmis- 
sion costs,  request  and  updating  of  the  files  and  a limit 
on  the  storage  capacity  of  each  computer.  The  model 
searches  for  the  minimum  of  a non-linear  zero-one  cost 
equation  which  can  be  reduced  to  a linear  zero-one  program- 
ming problem. 

Another  work  is  a paper  by  Casey  [9] . He  considers  a 
mathematical  model  of  an  information  network  of  n nodes,  some 
of  which  contain  copies  of  a given  data  file.  Using  a 
simple  linear  cost  model  for  the  network,  several  properties 
of  the  optimal  assignment  of  copies  of  the  file  are  demons- 
trated. One  set  of  results  expresses  bounds  on  the  number 
of  copies  of  the  file  that  should  be  included  in  the  net- 
work, as  a function  of  the  relative  volume  of  query  and 
update  traffic.  The  paper  also  derives  a test  useful  in 
determining  the  optimum  configuration. 

Of  very  recent  appearance  is  a paper  by  Mahmoud  and 
Riordon  ^22]  . In  this  paper  the  problems  of  file  allocation 
and  capacity  assignment  in  a fixed  topology  distributed 
computer  network  are  simultaneously  examined.  The  objective, 
in  that  analysis,  is  to  allocate  copies  of  information  files 
to  network  nodes  and  capacities  to  network  links  so  that  a 


minimum  cost  is  achieved  subject  to  network  delay  and  file 
availability  constraints.  The  deterministic  solution  for  a 
medium  size  problem  is  intractable  due  to  the  large  amount 
of  computation  so  that  an  heuristic  algorithm  is  proposed. 

A quite  different  analysis  in  which  the  important 
quantity  to  be  optimized  is  the  service  time,  instead  of  the 
operating  cost,  is  done  by  Belokrinitskaya  et  all.  [10]. 

The  analysis  results  in  a zero-one  nonlinear  programming 
problem  (that  can  be  linearized),  similar  to  the  one  in  [8]. 

In  the  above  mentioned  works,  the  problem  is  considered 
under  static  conditions  and  using  average  values  of  the 
parameters . 

If  the  parameters  of  the  system  are  time-varying, 
however,  or  if  the  exact  pattern  of  the  rates  of  demand  is 
unknown  or  some  non  negligible  possibility  of  node  or  link 
failures  is  expected,  then  some  kind  of  dynamic  approach 
must  be  used. 

It  has  been  only  recently  that  the  first  studies  of 
these  problems,  from  the  dynamic  point  of  view,  have  begun 
to  appear.  In  a work  by  A.  Segall  |ll]  the  problem  of 
finding  optimal  policies  for  dynamical  allocation  of  files 
in  a computer  network  that  works  under  time-varying  opera- 
ting conditions  is  studied.  The  problem  is  considered 
under  the  assumption  that  the  system  keeps  one  copy  of  each 
file  at  any  given  time.  The  case  when  the  rates  of  demand 


are  not  perfectly  known  in  advance  is  also  treated.  Only 
a prior  distribution  and  a statistical  behaviour  are  assumed, 
and  the  rates  have  to  be  estimated  on-line  from  the  incoming 
requests . 

The  problem  of  optimally  allocating  limited  resources, 
among  competing  processes,  using  a dynamic  programming 
approach  is  studied  in  fl2j  . A dynamic  programming  approach 
is  also  suggested  for  the  problem  of  minimizing  the  costs  of 
data  storage  and  accesses  in  [25]  . Here  two  different  types 
of  accessing  costs  are  considered.  The  accessing  cost  will 
depend  on  whether  a record  is  to  be  read  or  to  be  written 
(migration) . A different  approach  to  the  same  problem  is 
taken  in  [26]  . A two-node  network  with  unknown  access 
probabilities  is  considered.  The  problem  is  to  set  up  a 
sequential  test  which  determines  the  earliest  moment  at 
which  migration  leads  to  a lower  expected  cost. 

The  present  work  considers  the  problem  of  optimal 
dynamic  file  allocation  when  more  than  one  copy  are  allowed 
to  exist  in  the  system  at  any  given  time.  A general 
model  to  handle  this  problem  including  updating  traffic 
and  the  possibility  of  node  failures  will  be  developed. 

The  evolution  of  the  system  is  represented  as  a finite- 
state  Markov  process  and  dynamic  programming  will  be  used 
for  the  solution  of  the  optimization  problem. 
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1.2.  Summary  of  Results 

A model  for  the  analysis  of  optimal  dynamic  file  allo- 
cation is  introduced.  The  use  of  two  types  of  control 
variables,  one  for  adding  new  copies  to  the  system  and  the 
other  for  erasing  copies,  gives  to  the  model  certain  proper- 
ties that  permit  the  construction  of  an  efficient  and  rela- 
tively simple  algorithm  to  solve  the  optimization  problem. 
Among  others,  the  algorithm  is  efficient  due  to  the  fact  that 
it  computes  only  the  nonzero  transition  probabilities.  A 
detailed  set  of  flow-charts  and  Fortran  program  listings 
are  given  for  all  the  operations  and  calculations  that  take 
place  in  the  optimization  process. 

Within  the  same  framework  the  incorporation  of  node 
failures  presents  no  important  difficulties,  except  for 
increasing  the  number  of  states.  Some  kind  of  constraints 
in  the  state  space,  those  that  could  be  represented  as 
reductions  in  the  set  of  admissible  states,  are  also  easily 
handled  by  the  model . 

In  the  last  chanter  we  apply  the  algorithms  to  several 
numerical  examples.  For  the  case  of  constant  rates  of  demand 
with  no  failures  in  the  computers  the  corresponding  Markov 
•<  processes  have  a trapping  state.  For  these  processes  it  will 

be  shown  that  the  general  dynamic  programming  algorithm 
need  not  be  implemented,  and  a much  quicker  answer  to  the 
optimization  process  can  be  found. 

For  the  more  general  case  of  constant  parameters  with 
possibility  of  node  failures  included,  quick  convergence  to 
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the  steady  state  optimal  dynamic  decision  policy  was  found 
for  all  examples. 

Finally  it  will  be  shown  that  having  a completely 
simmetric  network  (equal  parameter  values  for  all  computers 
and  links)  will  allow  a considerable  reduction  in  the  number 
of  states. 

A more  detailed  exposure  of  results  can  also  be  found 
in  the  chapter  dedicated  to  "Conclusions  and  Open  Questions". 

1.3.  Chapter  Outline 

In  chapter  II  we  begin  with  the  description  of  the  model. 
We  first  state  the  general  hypothesis  and  basic  assumptions 
to  be  considered  throughout  the  study  and  continue  with  the 
description  of  the  operation  procedure.  We  indicate  the 
objective  function  and  define  the  control  and  allocation 
variables.  The  chapter  ends  with  the  definition  of  the 
state  and  the  description  of  the  dynamic  equations  of  the 
system. 

In  chapter  III  Stochastic  dynamic  programming  is  aoplied 
to  the  model  to  determine  the  optimal  allocation  strategy. 
First  we  will  write  the  recursive  equations  for  a simple  net- 
work with  only  two  computers  and  then  we  will  see  how  easily 
these  equations  generalize  to  any  number  of  computers.  We 
finish  the  chapter  indicating  how  the  model  can  handle  the 


framework  with  the  inclusion  of  the  uDdating  traffic  and  the 
possibility  of  node  failures.  As  in  chapter  III,  we  first 
write  the  recursive  equations  for  a network  with  two  computers 
and  then  generalize  them  to  any  number  of  computers.  At 
this  point  we  give  a very  detailed  set  of  flow-charts,  showing 
how  to  compute  the  different  matrices  and  vectors  of  the 
recursive  equations  and  how  to  carry  out  the  whole  optimi- 
zation process. 

Chapter  V deals  with  numerical  applications.  Using 
the  insight  gained  from  numerical  answers  some  additional 
analytical-  results  are  developed. 

A few  pages  dedicated  to  general  conclusions  and  further 
work  to  be  done  in  this  area  will  follow  this  chapter. 

Two  appendices,  A and  B,  expanding  results  of  chapters 
II  and  III  will  also  be  added.  A third  appendix  contains 
a set  of  Fortran  program  listings  corresponding  to  the  most 
significant  flow-charts  of  previous  chapters.  These  programs 
have  been  used  to  implement  the  numerical  applications  of 
chapters  V.  Auxiliary  subroutines  are  also  listed. 


CHAPTER  II 


DESCRIPTION  OF  THE  MODEL 


II . 1 Characteristics,  Basic  Assumptions  and  Operation 
Procedure 

We  shall  make  here  several  simplifying  assumptions, 
that  are  still  consistent  however  with  the  models  appearing 
in  real  networks.  We  shall  assume  that  the  files  are  reques- 
ted by  the  computers  according  to  mutually  independent 
processes  (with  statistics  to  be  specified  presently)  and 
also  that  the  files  are  sufficiently  short.  Moreover  the 
communication  lines  are  taken  to  have  sufficient  capacity 
and  the  computers  sufficient  memory,  so  that  the  transmission 
of  the  file  takes  a very  short  time  and  there  is  no  res- 
triction on  how  many  files  a computer  can  carry.  Under 
these  assumptions,  it  is  clear  that  in  fact  the  files  do 
not  interfere  with  each  other,  and  we  can  therefore  treat 
each  file  separately. 

The  analysis  will  be  done  in  discrete  time,  assuming 
the  existence  of  a central  synchronizing  clock.  It  will  be 
considered  that  with  previous  assumptions  the  time  interval 
between  clock  impulses  is  long  enough  to  allow  the  execution 
of  all  the  necessary  operations  to  take  place  in  it  (request 
arrivals,  "reading"  of  present  state,  implementation  of 
optimal  decisions,  etc.). 


In  this  chapter  the  possibility  of  node  failures  will 
not  be  included  in  the  model.  This  extension,  together  with 


the  inclusion  of  the  updating  traffic,  will  be  left  to 


We  may  summarize  the  assumptions  as  follows: 

1)  No  failures  in  the  network  (relaxed  in  Chapter  IV) 

2)  Channels  with  sufficient  capacity  (or  sufficiently 


3)  Sufficiently  large  storage  capacity  at  each  computer 

4)  Requesting  according  to  mutually  independent  processes 


assumptions  the  files  do  not  interfere  with  each 


The  proposed  procedure  is  similar  to  that  proposed  in 
reference  [ill  * with  the  only  difference  being  that  we  now 
allow  more  than  one  copy  at  each  instant  of  time  (the  way 


the  updating  traffic  is  taken  in  consideration  will  be 


described  as  follows:  Suppose  a certain  number  of  copies  is 


stored  at  time  t in  the  memories  of  a set  of  computers 


say  I.  If  at  time  t the  file  is  requested  only  by  computers 


in  the  set  I then  no  transmission  cost  is  incurred  and  a 


decision  has  to  be  made,  whether  to  erase  some  of  the  copies 
from  I (with  the  specification  of  the  particular  copies  to 
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Fig.  II. 1.  Illustration  of  the  Operation  Procedure 
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be  erased)  or  to  keep  the  same  number  of  copies.  If,  on 
the  other  hand,  the  file  is  also  requested  by  other  computers 
not  in  I,  then  the  file  is  transmitted  for  use  to  these 
computers  and  a new  collection  of  copies,  say  J,  appears 
in  the  system  at  time  t+.  A similar  decision  now  has  to 
be  made  but  with  the  set  J instead  of  the  set  I 

The  restriction  of  reallocating  the  file  only  in 
conjunction  with  a regular  transmission  is  reasonable  for 
this  model,  because  if  a change^ of  location  is  decided  upon, 
one  might  as  well  wait  until  the  file  is  requested  for  the 
next  time  by  the  appropriate  computer,  otherwise  it  is 
conceivable  that  the  file  might  be  transferred  back  and 
forth,  without  anybody  actually  using  it. 

II . 2 Data,  Parameters  and  Variables 

In  this  section  part  of  the  notation  used  in  the  study 
will  be  introduced. 

Consider  a completely  connected  network  of  NC  computers. 
The  requests  of  the  file  by  the  computer  will  be  modeled 
as  mutually  independent  Bernoulli  processes  with  rates 
9^(t),  i = 1,..NC,  that  is 

Pr{ni(t)  = 1}  = 1 ~ pr  fn±(t)  = 0}  = e±(t)  (2.1) 

i = 1, . .NC 


where  n^t)  = 1 indicates  that  the  file  has  been  requested 


yA(t)  = 


by  computer  i at  time  t.  The  rates  e^t)  are  assumed  to  be 
known  for  all  computers  and  instants  of  time. 

We  define  the  variables 

1 if  there  is  a copy  stored  at  computer  i at 
time  t (2.2) 

0 Otherwise 

i = 1, . .NC 

The  condition  of  having  at  least  one  copy  of  the  file 
in  the  system  at  any  instant  of  time  can  be  analytically 
expressed  as 
NC 

T;  y±(t)  >1  Vte[o,Tj  (2.3) 


i=l 


where  T is  the  whole  period  of  operation. 
The  operation  costs  are 


= storage  cost  per  unit  time  per  copy  at  memory  i 


= communication  cost  per  transmission  from  computer 


i to  computer  j 


i , j = 1 , . . . NC 


i f 3 


We  will  assume  C^  = 0 


Vi 


It  is  assumed  that  these  costs  are  time-invariant;  the  case 
with  time-varying  costs  can  be  handled  by  simply  writing 


C^  (t)  and  C^  (t)  throughout  the  paper. 


II . 3 Objective  Function 


Supposing  that  the  user  accesses  that  copy  of  the  file 
that  minimizes  his  communication  cost  and,  denoting  sim- 
bolically  by  I ( t ) the  set  of  nodes  having  a copy  at  time  t, 
we  can  write  the  expression  for  the  total  expected  cost  over 
the  period  |o,tJ  as 

T r NC  NC 


C = E 


E 


C y . (t)+ 
i i 


t=0  •-  i=l 


(1-y^ (t) ) ni (t)min  Cki 
i=l  kt  I ( t)  J 


(2.4) 


The  first  sum  in  the  bracket  represents  the  total  storage 
cost  at  time  t and  the  second  sum  is  the  total  transmission 
cost.  We  can  see  that  summands  contributing  to  the  trans- 
mission cost  are  those  with  yi(t)  = 0 and  n..  (t)  = 1 only,  that 
is,  those  coming  from  computers  that  do  not  have  the  file 
and  have  had  a request. 

The  goal  is  to  design  a closed-loop  control  that  will 
dynamically  assign  the  location  of  the  file  and  will  minimize 
the  defined  expected  cost.  We  introduce  the  control  variables 
in  the  next  section. 


I I . 4 Control  Variables  and  Restatement  of  the  Objective 
Function 

We  will  define  two  types  of  control  variables.  One  will 
correspond  to  the  erasure  process  and  the  other  one  to  the 
writing  process.  The  separation  of  these  two  operations  in 


the  amount  of  notation. 


ci(t)= 


The  variables  are 

1 — [if  the  decision  is  to  erase  the  copy  from  i 
at  time  t+  assuming  the  copy  was  there  at 
l=.  at  time  t (i.e.  y^t)  = 1)  (2.5) 


0 — otherwise 


a±(t)=( 


1 — [if  the  decision  is  to  keep  a copy  in  i at 

time  t+  assuming  that  the  copy  was  not  there 
(y.. (t)=0)  and  there  was  a request  from  that 


computer  (n.(t)  = 1) 

l 1 


(2.6) 


0 — otherwise 
i = 1, . .NC 

These  definitions  require  the  introduction  of  the  con- 
cept of  active  control  variables.  It  will  be  said  that  the 
variable  c^(t)  is  active  if  yi(t)  = 1 and  that  a^ft)  is 
active  if  yi(t)  = 0.  Due  to  these  definitions  a^(t)  and 
e^(t)  cannot  be  simultaneously  active.  From  definitions 
(2.5)  and  (2.6)  the  nonactive  variables  will  always  be 
equal  zero.  Therefore  only  active  variables  will  be  con- 
sidered throughout  the  analysis. 

With  the  previous  notations,  the  dynamic  evolution 
of  the  svstem  is: 
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Equation  (2.10b)  shows  that  if  our  decision  variables 
are  such  that  all  copies  of  the  file  will  be  erased,  then 
no  decision  variable  is  actually  implemented,  and  there- 
fore the  system  remains  in  the  previous  state.  Otherwise, 
the  system  evolves  according  to  equation  (2.10a)  namely 
computer  i will  have  a copy  at  time  (t+1)  if 
i)  it  had  a copy  at  time  t (y^(t)  = 1)  and  the  decision  was 

not  to  erase  it  (ci(t)  = 0 ) or 

ii)  it  did  not  have  a copy  at  time  t and  there  was  a request 

from  computer  i (n^ft)  = 1)  and  a decision  to  write  the 
file  into  memory  i was  taken  (c^ft)  = 1). 

The  optimization  problem  could  then  be  stated  as  follows: 
Given  the  dynamics  (2.10),  find  the  optimal  control  policies 


e J (t) 


a£  (t) 


t ~ 1,2,  . .T 


and  the  initial  locations  y^O),  Vi,  so  as  to  minimize  the 
expected  cost  (2.4). 

Hence  we  have  a dynamic  system  in  which  the  inputs  are 
a sequence  of  decisions  made  at  various  stages  of  the  evolu- 
tion of  the  process,  with  the  purpose  of  minimizing  a cost. 
These  processes  are  sometimes  called  multi-stage  decision 
processes  [15] . 


11.5  Definition  of  State  and  Dynamics  of  the  System 


/ 
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having  a zero  in  the  places  corresponding  to  computers  that 
do  not  have  the  file  and  a one  in  the  places  of  computers 
having  a copy.  These  vectors  will  be  named  by  the  decimal 
number  whose  binary  representation  is  the  NC-  dimensional 
vector  and  will  be  represented  by  a capital  Y. 

Therefore  the  state  at  time  t will  be  the  column  vector 


Y(t)  = 


yx(t) 

y-,  (t) 


yNC(t) 


(2.11) 


or  alternatively  the  state  of  the  system  at  time  t is  m(t) 
where 

m(t)  = decimal  number  with  binary  representation  given 
by  the  sequence  y1(t)  y2(t) — yNC(t) 

m l|2/...fM 

M = 2NC  - 1 

m = 0 will  not  be  a valid  state  because  it  corresponds  to 
the  case  of  having  no  copies  in  the  system  and  this  situation 
has  to  be  avoided.  Thus  the  previously  stated  condition 
NC 

2 yi(t)  - 1 
i = l 

is  tran' lated  to  this  notation  as  m ^ 0 


The  dynamics  of  the  state  are  easily  obtained  from  the 
dynamics  of  the  allocation  variables:  we  only  have  to 
substitute  for  each  component  of  the  vector  defining  the 
state 


(2.12a) 

iff  right  hand  side  of  (2.12a)  / 0 and 


y ( t+1 ) = Y ( t ) iff  right  hand  side  of  (2.12a)  = 


(2.12b) 


0 


To  clarify  further  these  ideas  let  us  write  the  ordered 
sequence  of  events  that  take  place  at  any  time  t 
- at  time  t~  the  optimal  e*(t)  and  o*(t) 
are  computed 


- at  time  t the  requests  arrive 

- at  time  t+  the  optimal  decisions  are  activated 
if  in  doing  so  the  system  does  not  go  to  state  0 
(that  is,  if  not  all  existing  copies  are  erased) . 
Otherwise  the  system  does  not  change  state. 

This  sequence  of  events  is  illustrated  in  Fig.  II-2 . 


Calculate  the  optimal 
e*(t)  and  a|(t)  given  Requests 

present  states  and  arrive 


Activate  the  optimal 
decisions  if  in  doing 
that  the  system  does 
not  go  to  state  0. 
Otherwise  do  not 
change  state. 
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u 1 ( Jl  f'2  ‘*3  £4  ' (0  ^ ® D 

Thus,  if  there  is  a request  from  computer  3,  the  system  will 
go  to  state 

yl  y2  y 3 y4  y5 ^ state  6 

0 0 110 


and  if  there  is  no  request  from  computer  3 the  system  will 
go  to 


1 2 3 4 5 

0 0 0 1 0 


state  2 


Because  of  the  unique  correspondence  in  the  notation  we  see 
that  it  is  equivalent  to  say  that  the  decision  is 
u = ( a1  e2  a3  e4  e5)  = (01101) 

or 

"go  to  state  6" 

For  the  sake  of  simplicity  these  two  forms  will  be 
interchangeably  used. 

From  the  above  analysis  it  can  br  seen  that 
(initial  state)  © (decision  vector)  = (final  desired  state) 
(decision  vector)  = (initial  state)  © (final  desired  state) 
where  © means  "exclusive  or".  This  is  so  because  if  a con- 
trol variable  has  a value  1 we  have  to  change  the  value  of 
the  allocation  variable  in  the  transition,  while  if  the  value 
is  0 there  is  no  change  in  the  transition.  This  kind  of 
operation  is  exactly  the  "exclusive  or"  addition.  In 


This  property  will  permit  to  easily  write  a tableau  showing 
the  transitions  among  states  assuming  the  requests  arrive 
from  all  computers. 

The  transition  tableau  for  the  case  NC=2is  shown  in  Fig.  II 
and  forNC  = 3in  Fig.  II-4  . 

For  simplicity,  the  place  corresponding  to  non-active 
variables  are  left  empty. 


Fig.  II-3.  Transition  tableau,  for  the  case  NC=? 

(assuming  deterministic  transitions) 


ion  tableau  for  the  case  JC=3 (assuming  deterministic  transitions) 
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These  tableaus  will  prove  later  to  be  of  great  utility 
for  the  construction  of  the  transition  probabilities  among 
states.  As  we  said  before,  we  can  write  this  tableau 
mechanically  and  this  is  important  in  computer  calculations. 

Summarizing,  for  a network  with  NC  computers,  the  steps 
are  the  following: 

1 - If  the  system  is  in  state  m,  write  m in  base  2 

with  NC  digits. 

2 - Assign  a control  variable  a to  the  places  where 

the  digit  is  0 and  a variable  e to  the  places 
where  the  digit  is  1. 

3 - To  obtain  the  value  of  these  control  variables  in 

a transition  from  m to  n,  compute  m ffi  n,  where  n 
is  also  written  in  base  2 with  NC  digits,  and 
assign  the  values  of  the  resulting  digits  to  the 
corresponding  control  variables. 

4 - To  obtain  the  mth  row  of  the  tableau  repeat  step  3 

NC 

for  values  of  n from  1 to  M (M  = 2 -1) 

5 - To  obtain  all  the  rows  of  the  tableau  repeat 

from  step  1 for  values  of  m from  1 to  M. 

The  flow-chart  corresponding  to  these  five  steps  is 
shown  in  fig.  II-5. 


Repeat  from  m = 1 to  M to  obtain 


all  the  rows  of  the  tableau 

Write  m in  base  2 with 
and  call  the  digits  ir^ 
i = 1, . « .NC 


if 

m 

is 

a 

if 

m 

is 

e. 

i = 1, . .NC 

Call  to  the  ith  variable  (c^ 
or  c±) 


Repeat  from  n = 1 to  M ^ 

to  obtain  all  the  elements  of  row  m 


Transition  to  state  n: 
write  n in  base  2 with  NC  digits 
Compute  m © n = k,  ith  digit 
= k i = 1, . .NC 


4 


Fig.  II- 5.  Flow  graph  showing  the  steps  to  obtain  the 
transition  tableau. 


CHAPTER  III 


DYNAMIC  PROGRAMMING  AND  BACKWARD  EQUATIONS 


III.1  Preliminary  Remarks 

It  can  be  easily  seen  that  the  model  described  in  chapter 
II  has  all  the  properties  needed  for  the  application  of 
dynamic  programming,  [13]  - [16]  . 

In  particular  it  is  obvious  that  the  separation  property 
holds  for  the  cost  function,  eq.  (2.4).  The  Markovian  state 
property  is  also  satisfied,  see  section  II. 5.  Hence  the 
problem  is: 

Given  the  dynamic  equations  (2.12),  find  the  optimal 
dynamic  allocation  strategy,  using  dynamic  programming,  to 
minimize  the  cost  (2.4). 

We  will  separate  the  total  expected  cost  (2.4)  in  two 


T-l 


parts 

C - E {H  [Y(T)]}  + E ^2  L[Y(t),t] 


(3.1) 


T = 0 


where 
L 


NC 


NC 


[Y  ( t ) , t]  = ^2  CiyiiT)  + ^2  (l-y1(T))ni(T)  min  Cki  (3.2) 


i=l 


i=l 


kel  (t) 


is  the  per  unit  time,  or  immediate,  cost,  and 
H [Y(T)]  = L [Y(T),T] 


(3.3) 


is  the  terminal  cost. 
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The  cost  to  go  at  time  t given  that  the  system  is  in 
state  i will  be  defined  as 

T 


ri(t)  = E j S LtY(T)'  t]  |v(t)  = i 


(3.4) 


T = t 


and  the  optimal  cost-to-go 

V*  (t)  = min  V . (t) 

1 u ( t ) 1 


i = 1,2, . .M 


(3.5) 


From  the  Markovian  property,  the  following  equalities 
can  be  easily  proved,  see  ref  £ll] . 

E (L  [y(t),t]  |Y(t),  t = 0,1,  ..t}  = 

E {L  [Y(t),t]  | Y (t)  } = (3.6) 


NC 


Z ciyi 


(t)  + E 


i=l 


NC 


y (l-y, (T) ) e.  (t)  min  C.  . | Y 

1 1 ktl(T)  kl  ' 


(T) 


i=l 


I I I . 2 Backwards  Recursive  Equations 

The  backwards  equations  for  this  probabilistic  system 
can  be  written  (see  [l^  pag  955)  as 

NC 

V*  ( t ) =min { E { L [*Y  ( t ) , tl  I Y ( t ) =i  } + Y''  P . . ( t , u)  V*  ( t+1 ) (3.7) 

1 u(t)  1 Z-r  i]  1 

3 = 1 

i = 1,2, . .M 

where  P^(t,u)  is  defined  as  the  probability  of  being  in 
state  j at  time  t+1  given  control  u and  given  that  the  system 


,rr 


is  in  state  i at  time  t,  that  is, 

(t,u)  = Prob  {Y (t+1)  = j | Y ( t ) = i,u(t)>  (3.8) 

From  the  expression  (3.2)  of  the  per  unit  time  cost  at 
time  t observe  that  the  decision  u(t)  at  time  t affects 
only  the  state  Y(t+1)  at  (t+1)  but  not  Y (t)  and  n(t)  and 
therefore  the  immediate  L(t)  cost  is  control  independent. 

If  u*  is  the  optimal  control  and  V*  the  corresponding 
cost  to  go,  then: 

NC 

V*(g)=E  {L  [Y(t),t]  | Y (t)  =i } + Pij <t,u*)V* (t+1)  (3.9) 

3 = 1 

i = 1,2,  . .M 


or  in  vector  form 


V*(t)  = A ( t)  + P(t,u*)  V*(t+1)  (3.10) 

With  this  notation  it  is  clear  that  the  total  minimum  expec- 
ted cost  over  the  period  [0,t|  will  be  the  smallest  component 
of  the  vector  V*(0)  and,  the  state  corresponding  to  this 
component  will  be  the  optimal  initial  state. 

To  pursue  further  with  the  investigation  of  the  actual 
form  of  the  vector  A ( t ) and  matrix  P(t,u*)  we  will  begin  with 
the  cases  NC=2  and  NC=3,  the  generalization  to  a larger 
number  of  computers  will  then  become  apparent. 


III. 3 Recursive  Equation  for  NC=2 

For  the  case  of  two  computers  the  expression  of  the 
total  expected  cost  over  the  period  ^3»t|  can  be  written  as 


C=E 


E 


t=0 


S Vi'M  * 2 (l-y1(t))n1(t)Cki 


i=l 


i=l 


k^i 


T 

= E S [ Ciyi(t)+C2y2(t)+(1"yl(t))0l(t)C21+(1'y2(t))02(t)C12] 

t=0  (3.11) 

where  we  have  applied  (3.6)  and  the  condition  that  y^t) 

+ y2<t)  _>  1.  Therefore 

L[Y(t)  ,t]=C1y1(t)+C2y2(t)  + (l-y1(t))01(t)C21+(l-y2  ( + ) )02(t)C12 

(3.12) 

From  this  expression  we  obtain  immediately  the  components  of 
the  vector  A(t) 

X1(t)=E{L[Y(t)  ,t]  |y  (t)  = (0  1)  =1}  = C2+C2101(t) 

A2  (t)=E{L[Y(t)  ,t]  |Y  (t)  = (1  0)  =2}  = C1+C1202(t)  (3.13) 

A 3 ( t ) =E { L [Y  ( t ) ,tj  |y  (t)  = (1  1)  =3}  = Cx+C2 

To  obtain  the  elements  of  the  probability  matrix  it  is 
very  important  to  follow  carefully  all  the  conditions,  see 
Fig.  II- 2,  imposed  on  the  decision  process.  Following  those 
rules  we  have  obtained  in  Appendix  A the  elements  of  the 


transition  matrix,  as 


l-a*(t)©1(t)  c* ( t)  a* (t)01 (t) (1-c* (t) )a* (t)91 (t) 
P(t,u*)  = e* (t)a* (t)02(t)  X-a*(t)02(t)  (l-e*(t) )a*(t)02(t)  (3.14 

e*(t)  (l-e*(t) ) (l-e*(t) )e*(t)  (1- e*  (t) ) (1-e*  (t) ) 

It  can  be  easily  seen  that  in  fact  P(t,u*)  can  be  obtained 
directly  from  the  tableau  of  Fig.  II-6  by  the  following  cor- 
respondence (see  Fig.  III.l): 

a)  If  the  value  of  the  control  variable  u^  in  the  tableau  is 
ui  = 0 write  a term  equal  to  1-u^ 


u.^  = 1 write  a term  equal  to  u^ 

b)  If  u.  = a-  write  a-8-  instead  of  a. 

l i li  l 

c)  If  the  cell  is  on  the  diagonal  add  to  the  previous  term 
a correcting  term  obtained  considering  a new  cell  with 
values  given  by  the  variables  y^(t)  and  y2(t)  and  applying 
steps  a)  and  b) 

d)  Repeat  a),  b)  and  c)  for  i=l  and  i=2.  The  transition 
probability  is  the  product  of  the  two  terms  obtained  in 


this  way. 


3=1  1 


0 

/ 


(l-e*(t))  (l-ajttje^tn  e*(t)a*(t)01(t)  (1-e*  (t) ) a*  (t)©x  (t) 

v c*(t)  (l-a*(t)6. (t)) 


Feting  term 


Fig.  III-l  How  to  obtain  the  first  row  of  the  matrix  from 
the  first  tableau  row 
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This  is  not  a surprising  result  and  it  could  be  easily 
expected  from  the  way  the  tableau  is  constructed.  Step  C) 
is  a consequence  of  the  condition  imposed  that  if  after  the 
arrival  of  the  requests  the  optimal  decision  requires  to  erase 
the  last  copy  of  the  system  we  remain  in  the  same  state. 
Therefore  the  probability  of  remaining  in  the  same  state 
(diagonal  terms)  has  to  be  corrected  by  a term  equal  to  the 
probability  of  requiring  the  erasure  of  the  last  copy.  This 
probability  is  exactly  the  probability  of  going  to  state  0 
if  this  state  were  allowed.  The  values  of  the  control  variables 
needed  to  go  to  state  0 are  obtained  through  the  "exclusive  or* 
addition  of  the  binary  representations  of  present  state  and  * 
state  0,  but  this  sum  is  always  equal  to  the  present  state 
representation;  therefore  the  values  of  the  control  variables 
are  equal  to  the  values  of  the  allocation  variables  of  the 
present  state.  In  this  way  we  ensure  that  this  matrix 
accomplishes  all  the  properties  of  a stochastic  matrix,  in 
particular  the  needed  condition  that  all  rows  must  add  to  one; 
this  is  so  because  the  terms  are  obtained  using  all  possible 
combinations  of  0's  and  l's  with  two  elements  (NC  elements 
in  general)  and  hence  we  always  add  terms  like 

A B+ ( 1-A) B+A (1-B) + ( 1-A) (1-B)  = 1 (3.15) 

Another  simplification  can  be  obtained  by  observing 
that  in  every  row  the  combination  of  control  values  that  will 
take  the  system  into  the  state  Y(t+1)=0  is  not  allowed.  For 
example  in  the  first  row  of  Fig.  II-3,  a1  = 0 r.2=l  is  forbidden 


- 


1 


i 
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and  therefore  in  the  first  row  of  P(ti u)  we  have 

(l-ai)e2  = 0 (3.16c 

Similarly  in  the  second  row  of  P(t,u) 

(l-a^ej^  = 0 (3.161 

and  in  the  third  row 

e2  = 0 (3.16c 

This  property  will  be  useful  sometime  to  simplify  the 
expression  of  the  transition  probabilities.  We  have  made  use 
of  this  property  in  Appendix  A. 

Grouping  all  results  together  we  obtain  the  following 
backward  matrix  equation  (NC  = 2) 


V*  (t) 


V*  (t) 


C2  + c21e1(t) 


v*(t)  = c,  + c. ( t ) + 


12  2 


c + c 

v_i  c.2 


1-a? (t) 0. (t)  c* (t) a? (t)9. (t)  (l-ei(t)o?(t)01(t)  V*(t+1) 


e*(t)a*(t)92(t)  l-a*(t)02(t)  (1~e*  (t)  a*  (t)  02  (t)  V*(t+1) 


e*(t) 


e*(t) 


l-e*(t)-e*(t) 


V*(t+1) 


(3.17) 


where  the  optimal  decisions  for  each  row  of  the  matrix 


P (t, u*)  are  the  values  of  the  corresponding  row  in  the  tableau 
that  give  minimum  scalar  product  with  the  vector  V* (t  + 1) 

In  particular,  if  we  define 


A=V* (t+1) 

B=(l-0,  (t)  )V?(t+l)+0.  ( t ) V*  ( t+1 ) 


(3.18) 


C=(l-01(t)  )V*  (t+D+S^tJV*  (t+1) 


then  if  the  system  is  in  state  1 at  time  t 
(a*(t)  = 0 

J if  A < B and  A < C (3.19a) 

( ( t)  = 0 


a*(t)  = 1 

if  B < A and  B < C (3.19b) 

(t)  = 1 


a*(t)  = 1 

if  C < A and  C < B (3.19c) 

e*(t)  = 0 


In  the  same  way  the  optimal  decisions  being  in  state  2 
and  3 can  be  obtained. 


We  will  see  some  numerical  applications  of  these  equa- 
tions in  chapter  V. 


III-4.  Recursive  Equations  for  NC=  3 and  Generalization  to 
any  NC 

For  NC=3  the  total  expected  cost  over  the  period 
can  be  expressed  as  (remember  = 0,  Vi) 


C 


E 


£ 


£ 


3 

ci*i(t)+  £ 
i=l 


£ £ Yi (t) y^ (t)nk (t)minCik 


3>i 


k*i 

k^j 


T 

"22  L[Y(t),t] 


E 


(3.20) 
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The  components  of  the  vector  A(t)  are  obtained  as 

X1(t)=E  (L[Y(t),t]  I Y (t)  = ( 0 0 1)  = 1 } * C3+C3161(t)+C3202(t) 

X2(t)=E  {L[Y(t)»t]  |Y(t)=(0  1 0)  = 2}  = C2+C2101(t)+C2303(t) 

A 3 ( t ) =E  {L[Y(t)#t]  | Y ( t ) = ( 0 1 1)  = 3}  = C2+C3+0L(t)min{C21,C31} 


H M M 


etc. 


The  whole  vector  is 


31 

0x(t) 

+ C32 

©2  (t) 

21 

&1  ( t) 

+ C23 

©3  (t) 

3 + 

Q1  (t) 

min 

(C21 ' 

C31 

12 

02  (t) 

+ C13 

©3(t) 

3 + 

©2  (t) 

min 

(C12 ' 

C32 

2 + 

©3  ( t) 

min 

(C13' 

C23 

^ 1 + C2  + C3  j 

The  way  to  construct  these  components  from  the  state 
vector  is  simple. 

The  easy  rules  are  sketched  in  the  flow  chart  of  Fig. 
III-2 . 

iRepeat  from  m=l  to  M 

! ! 

State  m 

write  m in  base  2 with  NC  components 
iik  ith  component  i=l,..NC 


Fig.  III.  2 Flow  chart  

to  obtain  the  per  J 
unit  time  cost  j 
vector  U_ 


set  of  indexes  such  that  nr  = 0 
set  of  indexes  such  that  mi=l 


!t)=  Z C.+  X)©s<t)min  {C,~ 
m i.I  1 j«J  ^ i«I  ^ 


1 
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As  far  as  the  probability  transition  matrix  is  concerned, 
we  can  calculate  easily  its  components  by  making  use  of  the 
rules  stated  for  the  case  NC=2  and  the  tableau  of  Fig.  II-7. 
Some  of  the  components  are  shown  below  (for  brevity  we  delete 
the  variable  t) . 

P(t,u)  = 


where  we  have  applied  a property  similar  to  (3.16)  so  that 
for  example  in  the  7th  row  of  P given  in  (3.23)  we  have 

£l€2e3  = °* 

It  can  be  seen  now  that  the  rules  we  developed  in  cons- 
tructing the  immediate  cost  vector  and  the  transition  pro- 
bability matrix  for  NC=2  and  NC=3,  generalize  easily  for  a 
network  of  arbitrary  size.  These  rules  will  allow  for  an 
easy  algorithm  to  be  implemented  on  a computer.  To  make  things 
concrete,  we  illustrate  this  in  the  following  example: 

Example: 

Suppose  we  have  a network  with  five  computers  NC=5,  and 
being  in  state  3 at  time  t,  we  want  to  know  the  immediate 
cost  and  the  probability  of  being  in  state  17  at  time  t+1. 

First  of  all  we  write  the  vector  representation  of  state 


(1-a^©^) (l-a2©2^  ^~ai®l^ a2°2e3 


(l*ai6l)  e2a3®3  (l-°1®1Ml-03®3) 

(1-a.e, ) e. (1-e,)  (l-a.0. ) (1-e.) e 


(3.23) 


11'  2 


r'l' 


2*  3 ‘ 


v2 


e e 

13 


— 


3 and  its  control  variables: 
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Y ( t ) = 3 = (0  0 0 1 1) 


(otj^  r (^2  1 9 £5) 


(3.24) 


From  this  representation  we  can  immediately  write  the 
per  unit  time  cost. 

A3(t)  = C4+C5+01(t)  min  (C41,  C51)  + ©2(t)  min  (C42,  C52) 

+e3(t)  min  (C43»  C53)  (3. 


(3.25) 


To  obtain  P,  ._(t)  we  also  need  the  vector  representation 
3^17 


of  state  17 

Y (t+1)  = 17  = (10001) 


(3.26) 


the  value  of  the  control  variables  we  need  for  this  transi- 
tion are: 

(0  0 0 1 1)  ® (1  0 0 0 1)  = (1  0 0 1 0)  = (a1a2a3e4e5)  (3.27) 


therefore 

3 

(0  0 0 1 1) 


al  a2  °3  e4  £5 


10  0 10 


(10001)  (3.28) 


now  we  can  write  that  the  transition  probability  is 
P3,17=0liei  (1-a292)(1“Ql3e3)  G4  (1"e5) 


(3.29) 


It  can  be  useful  to  verify  that  in  fact  we  will  arrive 
to  the  same  expression  if  this  probability  is  computed  by  a 
straight  forward  calculation. 

From  the  discussion  in  section  II-6  and  Fig.  II- 2 we 
see  that  we  can  begin  in  state  3 and  finish  in  state  17  in 
four  different  ways: 

1)  We  decide  to  go  to  (1  0 0 0 1)  = 17  and  there  is  a request 
from  computer  1 
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decision 


1 0 0 1 


Prob  { Pos.  1 } * (l-a2)  (l-a3)e4  (1-e^)  Prob  (n^-l) 


=a1(l-a2)  (l-a3)  (l-eg)©^^ 


(3.30) 


2)  We  decide  to  go  to  (1100  1)=25  and  there  is  a request 
from  computer  1 but  there  is  no  request  from  computer  2 
decision  = (11010) 

Prob  { Pos.  2 } *ot  a (1-a  ) c (1-e  )0  (1-0  ) (3.31) 

1 2 3 4 5 1 2 

3)  We  decide  to  go  to  (1  0 1 0 1)  = 21  and  there  is  no  request 

from  computer  3 but  we  have  a request  from  computer  1: 
decision  = (10110)  (3.32) 

Prob  { Pos.  3}  =0^  (l-a2)  a3£4  (l-e5)  (l-©3) 

4)  We  decide  to  go  to  (11101)  =29  but  there  is  no  request 
from  2 and  3 and  we  have  request  from  1 

decision  = (11110) 

Prob{Pos . 4 } = a1ot2o3e4  (l-£5)01(l-e2)  (1-0  ) (3.33) 

As  we  can  see  these  four  possibilities  are  the  results 
of  the  following  four  decisions 


(3.34) 


al 

a2 

a3 

e4 

E 5 

1 

0 

0 

1 

0 

with 

prob. 

1 

1 

0 

1 

0 

with 

prob. 

1 

0 

1 

1 

0 

with 

prob. 

1 

1 

1 

1 

0 

with 

prob . 

Adding  up  those  four  probabilities  we  have 
Prob{Pos . 1 }+  Prob{Pos . 2}  + Prob{Pos.  3}  + Prob{P os.  4} 

= (l-a2)  (l-a3)  e4  (l-e5)  0-j^  + 

+ a3  a2  (l-a3)  e4  (l-£5>  + 

+ (l-a2)  a3  e4  ( l-e5) ©^ ( l-©3)  + 

+ a2  a3  e4  (l-£^)93 (l-©2) (l-©3)  = 

=aieie4  [ (l-a2)  (l-a3)  + a2(l-a3)  (l-©2)  + 

+(l-a2)  a3  (l-©3)  + a2  a3  (l-02) (1~03)J  = 
=a3©3£4  (1— £^)  (l~ot2©2)  (l~ot3©3)  = ?3  (3.35) 

as  was  obtained  in  (3.29) 

We  could  have  written  the  remaining  probabilities  in 
the  transition  matrix  in  the  same  way  as  we  did  for  p3>17* 
Therefore  in  order  to  analyze  any  network  under  the  conditions 
stated  in  chapters  I and  II  we  only  have  to  build  up  the 
recursive  equations,  using  the  rules  described  before  and 
move  backward  in  time  until  we  reach  the  steady  state,  or 
arrive  at  t=0. 

Nevertheless  while  implementing  the  dynamic  programming 
procedure  we  do  not  need  to  calculate  all  the  probabilities 
of  the  transition  matrix.  As  it  will  be  seen  below  the 
reason  is  that  after  the  control  values  are  decided  upon 
many  of  the  terms  will  be  known  to  be  zero.  For  instance, 
consider  the  case  of  the  above  example,  in  which  we  were  in 
state  3 and  the  decision  was  "go  to  state  17”.  The  only 


probabilities  that  will  be  different  from  0 in  the  3rd  row  of 
the  transition  matrix  are  P3  17  and  P3 where 

3 = (0  0 0 1 1) 

u = (1  0 0 1 0)  = (a1a2a3e^e5)  (3.36) 

17  = (1  0 0 0 1) 

1 = (0  0 0 0 1) 

The  reason  is  that  the  only  condition  needed  to  accomplish 
the  decision  is  having  a request  from  computer  1,  the  only 
computer  in  the  decision  vector  with  a control  variable  a equal 
to  1.  If  this  request  does  not  come  the  system  will  move  to 
state  1 (we  only  excute  e4  = 1)  and  there  is  no  possibility 
to  go  to  any  other  state  with  that  decision  vector. 

This  rule  can  be  easily  generalized.  Being  in  state  n and 
having  made  decision  u(t),  the  only  probabilities  different 
from  zero  in  row  n of  P(t,u)  are  the  probabilities  corres- 
ponding to  destination  states  resulting  from  applying  to 
state  vector  n the  decision  vectors  obtained  from  vector 
u(t)  making  all  possible  substitutions  of  O's  and  l's  in 
places  where  there  are  copying  variables  (a's)  equal  to  1 in 
u (t)  . 

For  instance,  if  n=3  as  before,  but  now  u(t)  = (11010) 
we  will  have 

3 

(0  0 0 1 1) 


aI^2a3c4r-5 
110  10 


25 

(11001)  (3.37) 
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Then,  in  order  to  implement  this  decision, requests  from 
computers  1 and  2 are  needed,  and  therefore  we  have  the 
following  possibilities 


♦ (1  1 0 0 1)  = 25 
with  Prob 

♦ (1  0 0 0 1)  = 17 
with  Prob  ©^(l-©2) 

♦ (0  1 0 0 1)  = 9 
with  Prob  (1-©1)©2 

*-  (0  0 0 0 1)  = 1 

with  Prob  (1-©^) (l-©2) 

If  would  have  had  che  value  1 instead  of  0 the  last 
transition  will  go  to  state  3,  the  starting  state,  in  order 
to  avoid  the  erasure  of  the  last  copy. 

A schematic  way  showing  how  to  compute  the  transition 
probabilities  using  these  rules  is  shown  in  Fig.  II-3.  A 
flow-chart  showing  how  to  compute  row  n,  of  the  probability 
transition  matrix,  when  the  decision  is  "go  to  state  m" , is 
shown  in  Fig.  III. 3 b) . In  the  flow-chart  we  assume  that 
we  have  available  a subroutine  called  BITS  such  that  given 
n,  a number,  and  NC  number  of  components  it  returns  the  base 
2 representation  of  n with  NC  components.  The  calling 
sequence  will  be 


i 


* 


5 


CALL  BITS  (n,  NC,  NB2 ) 

Furthermore,  we  assume  we  also  have  available  the 
function 

L = DECI (k,  LBk , NC) 

such  that  given  a vector  LBk  with  NC  components  it  returns 
the  number  L whose  representation  in  base  k is  LBk. 

These  subroutines  are  given  in  Appendix  C. 

The  simplifications  explained  so  far  can  produce  a 
great  saving  in  computation  because,  for  instance,  in  the 
first  case  presented,  only  2 of  the  2^-1  = 31  components  are 
different  from  zero. 

The  optimization  procedure  for  each  starting  state  will 
consist  then  in  the  computation  of  the  non  null  probabili- 
ties for  the  initial  state  row  for  every  possible  transition; 
taking  scalar  product  of  these  non  null  probabilities  by 
the  corresponding  costs  - to  - go  and  choosing  the  smallest 
result.  The  decision  giving  place  to  the  smallest  product  is 
the  optimal  decision  for  that  initial  state  and  the  product 
added  to  the  per  unit  time  cost  for  that  state  will  produce 
the  next  (backward)  cost  - to  - go. 

The  flow  chart  of  fig.  III-4  shows  the  set  of  operations 
involved  in  the  optimization  process.  In  the  next  section  «• 

I 

we  will  show  how  this  model  can  be  easily  extended  to  problem^ 
with  constraints  in  the  state  space. 


51 


Initial  State  = n 
Decision  "Go  to  state  m" 

Write  n and  m in  base  2 with  NC  digits 
n.,  m . are  the  iths  digits  respectively 


Decision  vector  u(t)  = n © m 

Decision  variables  u.(t)  ni  _ 0 

i (t)  if  n.  = 1 


Call  I to  the  set  of  subindeces  such  that 
u^  ( t ) = ouft)  (i.e.  n^  = 0)  and  oi^t)  * 1 

NI  = number  of  elements  in  I 


Form  the  decision  vectors  u (t)  where 

..  _ , .NI  


v = 1,  . .2L 


according  to 


uv  (t)  - |Ui<t)  - 0 and  a.  - 


vth  combination  of^  0 ' s and  l's 
in  places  where  ni  = 1 and  0^  = 0 


1 = n © uv(t)  L ={1}  , 2N  different  l's 
The  non  null  probabilities  in  row  n when 
decision  is  "go  to  state  m”  are 

Pnl  (t)  = j 0j(t)  k (1  ‘ 0k(t))  V1<L 
where  j t I and  = 1 in  uv(t) 

k * I and  = 0 in  uv(t) 


Fig.  III-3  a)  Flow-chart  showing  how  to  construct  the 

non-null  probabilities  of  row  n when  decision 
is  "go  to  state  m" 


*The  first  combination  is  0O---O  and  the  last  one  is  111  — 1 


m/JTa 


IA  = NAUX(k) 

LB2 ( IA)  = JB2 (k) 

A = A*  [(l-JB2(k) ) (1-0  ) + JB2(k)0  ] 
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III-5.  Constraints  in  the  State  Space 

The  problem  formulation  and  the  model  described  will 
allow  us  to  handle  easily  constraints  in  the  state  space. 
These  constraints  may  take  the  form  of  a maximum  number  of 
copies  allowed  in  the  system  at  any  instant  of  time  or, 
not  allowing  copies  of  the  file  simultaneously  in  two  or 
more  given  computers. 

For  instance,  if  given  a network  with  three  computers 
(NC=3) , three  copies  are  not  allowed  in  the  system  simul- 
taneously then  state  7 will  be  taken  out  of  the  set  of 
admissible  states;  if  on  the  other  hand,  the  restriction  is 
that  there  cannot  be  copies  simultaneously  in  computers  1 
and  2,  then  state  6 is  taken  out  of  the  state  space. 

One  example  of  these  types  of  constraints  was  presented 
before  when  state  0 was  not  allowed.  Therefore,  unallowed 
states  will  be  treated  here  in  the  same  way  state  0 was 
treated  before.  To  gain  some  i<  sight  we  present  an  example: 
Consider  a network  with  four  computers  (NC=4) . If  the 
present  state  is  1 = (0001)  and  the  decision  is 
u(t)  = (1110),  then  the  intended  state  is  15  = (1111). 


1 

(0  0 0 1) 


al 

a2 

*3 

C4 

1 

1 

1 

0 

If  there  is  a request  from  computers  1 and  3 but  not 
from  computer  2,  the  system  will  go  to  state  (1  0 1 1)  = 11 
and  this  event  will  occur  with  probability  G^U-e^e^. 


•UK  T*  - 
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Suppose  now  that  state  11  is  not  allowed,  then  another 
decision  has  to  be  made.  The  situation  is  such  that  any 
state  of  the  form  (a  o b c)  can  be  reached  where  a,  b and  c 
can  be  0 or  1 (but  not  all  three  equal  to  0) . Nevertheless, 
considering  so  general  a decision  at  this  point  will  make 
the  problem  very  complicated,  therefore  a decision  to  remain 
in  the  same  state  will  be  considered.  This  is  a particular 
case  of  the  whole  set  of  possible  decisions.  This  kind 
of  decision  will  give  to  state  11  the  same  treatment  as 
to  state  0,  as  suggested,  before.  The  probability  of  remaining 
in  one  state  will  now  be  composed  of  the  following  terms: 
Prob{remain  in  same  state}=  Prob{going  to  this  state}+ 

+ Prob{going  to  state  0}  + Probfgoing  to  not  allowed  states} 

(3.39) 

In  the  general  algorithm  we  will  eliminate  the  rows 
and  columns  corresponding  to  unallowed  states  and  add  their 
probabilities  to  the  diagonal  terms.  It  should  be  noticed 
that  some  extended  simplification  properties,  similar  to  the 
one  obtained  in  (3.16),  could  be  obtained  from  the  new 
unallowed  states. 

For  instance,  for  the  example  above  where  state  1 was 
the  current  state  and  state  11  was  not  allowed,  we  have  that 
(l-a2)a3 (l-c4)  = 0 (3.40) 

if,  moreover,  state  (1  0 0 1)  = 9 were  not  allowed  as  well, 

a1 (l-a2)  (l-a3)  (l-e4)  = 0 (3.41) 


then 


and  from  these  two  equalities 

c^U-c^)  (l-e4)  = 0 (3.42) 

An  illustrative  example  where  these  facts  are  applied  is 
studied  in  Appendix  B for  the  case  of  a network  with  two 
computers  (NC=2)  . It  is  shown  there  that  if  we  restrict 
the  system  to  have  only  one  copy  at  any  instant  of  time  the 
backward  equations  simplify  to  the  equations  given  by  A. 
Segall  in  L»].  where  the  restriction  was  to  operate  with 


only  one  copy. 


CHAPTER  IV 


UPDATING  TRAFFIC  AND  NODE  FAILURES 


IV-1.  Updating  Traffic 

The  updating  traffic  consists  of  requests  generated  at 
some  nodes  after  a request  of  the  file,  with  the  only 
purpose  of  modifying,  partially  or  completely,  the  content 
of  the  file.  With  this  definition  it  is  seen  that  the  updat- 
ing information  generated  at  any  node,  should  be  sent  to 
all  other  nodes  that  possess  a copy  of  the  file. 

It  will  be  assumed,  for  the  present  study,  that, 

1)  This  kind  of  traffic  is  generated  at  any  node  as 

a fraction,  of  the  query  traffic  of  this  particular 
node.  In  general  these  fractions  can  be  time  depen- 
dent variables.  If  we  denote  them  by 
p 1 (t),  p 2 (t),  - - p NC  (t) 

the  rate  of  updating  traffic  generated  from  node 
i at  time  t will  be  then 

p^tie.ft) 

2)  The  updating  traffic  is  implemented  before  the 
decision  has  been  activated  but  after  the  request 
has  taken  place.  The  sequence  of  events  is  represen- 
ted in  fig.  IV-1  as  a generalization  of  fig.  II-2. 


A 
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Computation 

Requests 

Implementation 

Activation  of 

of  optimal 

come 

of  generated 

optimal  decisions 

controls 

updating 

traffic 

r p 

Fig.  IV-1.  Sequence  of  events  including  updating  traffic 


I 


With  probability  p^(t)  an  updating  of  the  file  is 
generated  at  any  computer  i that  requested  the 
file  and  is  sent  to  all  computers  that  will  keep  a 
copy  at  time  t+1. 

3)  We  will  assume  that  there  is  no  conflict  between 
the  updating  commands  coming  from  different  com- 
puters. This  is  sometimes  a serious  problem  in  a 
practical  case  because  it  can  force  us  to  block 
requests,  while  some  updating  is  being# done  in  order 
to  avoid  the  processing  of  some  old,  and  then  use- 
less or  even  conflictive,  information. 

Under  assumptions  1)  and  2)  we  can  say  that  updating 
traffic  is  not  a function  of  present  state,  but 
only  of  present  rates  and  subsequent  states  (as  we 
will  see  in  sections  IV-2  and  IV-3,  this  property  is 
not  true  if  we  include  the  possibility  of  node 
failures) . The  only  change  to  take  place  in  the 
recursive  equations  will  be  in  vector  V*(t+1)  that 
will  have  some  extra  terms  added  to  its  components. 
The  new  costs-to-go  vector  at  time  t+l  will  be  now 


- 


[ 


/ 


UM 


i ... — .....  i mum  . , — i i 


N- 1 


v*(t+i)  + J2  Pk(t)  em(t)  ckN 

k==  1 


N 

V*(t+1)  + £ Pk<t,ek(t)  Ck,N-l 
k=l 
k^N-1 

N 

vj(t+i)  + E ck(t|9k(t)  E cki 

k=l  1 « I ( j ) 

1/k 


N 

°k(t)  ek(tl  E cki 
1=1 
l^k 

Here  I ( j ) is  the  computers  containing  a copy  at 
state  j or  in  other  words  the  set  of  subscripts 
corresponding  to  l's  in  vector  state  j.  We  are 
assuming  here  that  the  only  charge  involved  in  up- 
dating a copy  at  computer  i by  computer  j is  the 
transmission  cost  C^. 

The  recursive  equation  will  now  be 

V*(t)  = A (t)  + P(t,U*)  |\*(t+l)  +R(t)  J (4.2) 


N 

VM(t+1)  + £ 
k=l 


=V* (t+1) +R (t) 
(4.1) 


zsni 
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where  R(t)  is  an  M-components  column  vector.  Several 
examples  investigating  how  the  updating  traffic  affects 
the  dynamic  control  of  the  system  will  be  studied  in 
Chapter  V. 

IV-2 . Network  with  Node  Failures 

In  this  section  the  necessary  changes  in  the  model 
to  include  the  problem  of  node  failures  will  be  considered. 

We  shall  assign  to  every  computer  a probability,  Pf,  of 
failure  and  a probability,  Pr,  of  recovery  according  to  the 
following  definitions 

P^AProb.  of  failure  per  unit  of  time  (4.3) 

P =Prob.  of  recovery  per  unit  of  time  given 
r that  the  computer  is  out  of  order 

It  is  assumed  that  P^  and  Pr  are  independent  and  the 
same  for  all  computers  and  instant  of  time,  or  in  other 
words  that  the  failure  and  recovery  processes  are  modeled 
as  two  independent  Bernoulli  processes  with  rates  Pf  and 
respectively. 

Under  these  circumstances,  the  new  state  has  to  carry 
along  information  about  two  facts 

1)  the  computer  condition  (working  or  not)  and, 

2)  if  the  computer  is  on,  whether  or  not  the  computer 
has  a copy  of  the  file 


S>t* 
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Therefore  it  can  be  concluded  that  each  component  of  the 
vector  state  has  to  bring  information  of  one  of  three 
possibilities 

a)  computer  out  of  order 

b)  computer  working  and  without  a copy  of  the  file 

c)  computer  working  and  with  a copy  of  the  file 

If  we  represent  possibilities  b)  and  c)  with  a 0 and 
1 respectively,  as  before,  and  possibility  a)  with  the 
digit  2,  the  new  state  vector  will  be  equivalent  to  a base 
3 representation  of  some  decimal  number.  Since  this  repre- 
sentation is  unique  we  see  that  we  can  name  the  states  by 
the  decimal  number  whose  base  3 representation  is  the  NC-dimen- 
sional  vector.  For  example 

Y ( t ) = (0211) 

2 

will  correspond  to  the  state  l+3+2x3  =22  of  a network 

with  4 computers,  where  computer  2 is  out  of  work,  computers 
1,  3 and  4 are  working  and  the  last  two  have  a copy  of  the 
file  in  their  memories. 

Our  model  will  now  contain  the  following  further  assump- 
tions : 

a)  When  a computer  is  restored,  it  comes  up  with  no  copy 
in  its  memory.  This  says  that  no  computer  can  make 

a transition  from  state  2 to  state  1. 

b)  If  there  is  at  least  one  computer,  say  i,  in  working 
condition  but  there  are  no  copies  in  the  system  then 
one  copy  is  brought  from  outside  (special  memory) 


. . ' V'SJK* 


at  a price  Cq^.  If  there  are  several  computers  in 


working  condition  with  no  copy  in  their  memories, 
the  system  will  bring  a copy  from  outside  to  the 


computer  with  the  smallest  CQ^.  Obviously 


C . >>  C . . 

01  1] 


i,  j = 1,2, . .Nr 


and  the  quantities  CQi  will  carry  a measure  of  the 


risk  of  losing  all  copies  that  we  are  willing  to  take 

+ 


c)  The  time  between  the  points  t and  t of  Fig.  II-2 
is  very  small  compared  to  the  unit  interval  (t,  t+1) . 


Por  dUf  purpose  that  means  that  the  probability  of 

+ , 


a failure  in  the  interval  (t  , t ) is  negligible. 

With  these  assumptions  the  number  of  states  in  the  state 


space  will  be  the  number  of  different  NC-dimensional  vectors 

NC 

that  can  be  formed  with  3 digits,  that  is,  M=3 

In  the  present  case  state  0 is  in  the  state  space, 
because  the  system  can  go  to  this  state  after  being  at 
state  M-l,  when  all  the  computers  are  inoperative,  provided 
that  all  computers  become  operative  in  only  one  interval  of 
time.  The  decision  variables  will  remain  the  same  as  before 
except  that  there  are  no  decision  variables  for  inoperative 
computers.  That  is,  there  are  not  decision  variables  for 
the  components  of  the  state  vector  with  value  equal  to  2. 

In  particular,  when  all  computers  are  not  operative,  there 
is  no  decision  to  be  made.  The  only  thing  to  do  is  to 
wait  until  one  or  more  computers  recover  and  then  bring  a 
copy  into  the  system  from  outside. 


Given  the  state  at  time  t,  the  transition  to  time  (t+1) 


will  be  obtained  as  follows 

a)  Decide  upon  the  value  of  the  control  variables 
(if  any) 

b)  Perform  "exclusive  or"  of  the  control  variables 
with  the  actual  requests  and  modify  accordingly  the 
state  variables  (as  in  section  II-6) 

c)  The  failure  or  recovery  of  computers  (if  any)  will 
modify  in  turn  the  former  transition. 

The  state  at  time  (t+1)  will  then  be  the  result  of  the 
above  three  operations. 

In  following  sections  we  apply  these  concepts  to  the 
case  NC=2 , NC=3  and  show  how  they  generalize  to  any  NC. 


IV. 3.  Recursive  Equations  for  NC=2  considering  Node 
Failures 

The  states  per  NC=2  are 


Y(t)  0=  (0 
1=  (0 
2=  (0 
3=  (1 
4=  (1 


0) 

1) 

2) 

0) 

1) 


Y (t) =5= ( 1 2) 
6= ( 2 0) 
7= ( 2 1) 
8=  (2  2) 


(4.4 


Let  be  the  costs  of  bringing  copies  from  outside 

to  computers  1 and  2 respectively.  Assuming  < CQ2 

the  per-unit-time  costs  are 


*0(t) 

C01 

x5(t) 

a:  (t) 

= C2  + 

c21e1(t) 

Vfc) 

a2  (t) 

C01 

a7  (t) 

A,  (t) 

= c,  + 

c,,e,(t) 

AQ(t) 

'1 

'02 

'2 


(4.5 
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It  seems,  fre">  these  values  of  the  costs,  that  the  optimi- 
zation process  will  try  to  keep  the  system  always  at  state 
8,  because  this  state  has  the  smallest  cost,  but  the  decision 
"go  to  state  8"  is  not  among  the  set  of  admissible  decisions, 
since  this  will  erase  all  copies  from  the  system. 

For  the  case  NC=2  the  only  states  that  will  generate 
control  variables  are  states  1,  3 and  4 and  clearly  these 
control  variables  will  give  rise  to  transitions  am  ong 


these  states  only 


If  we  represent  by: 

B the  transitions  ruled  by  control  variables 

(when  no  failures  or  recoveries  are  involved) 

X the  transitions  due  to  some  failure  or  recovery 

of  some  computer 

* the  transitions  due  to  a forced  decision 

(namely  a copy  has  to  be  brought  from  outside) 


The  following  tableau  of  possible  transitions  can  be 
sketched  (remember  C, 


'01 


C02) 


States 


states 


0 1 

2 

3 

4 

5 

6 

7 

8 

* 

X 

X 

X 

a 

X 

a 

a 

X 

X 

X 

X 

X 

★ 

X 

X 

a 

X 

a 

a 

X 

X 

X 

X 

a 

X 

a 

a 

X 

X 

X 

X 

X 

★ 

X 

X 

X 

X 

* 

X 

X 

X 

* 

X 

X 

X 

X 

X 

S 

t 


a 

Comps 

t 

e 

1 2 

0 

0 0 

1 

0 1 

2 

0 2 

3 

1 0 

4 

1 1 

5 

1 2 

6 

2 0 

7 

2 1 

8 

2 2 

(4.6) 
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The  empty  entries  in  the  tableau  correspond  to  impos- 
sible transitions;  this  is  an  important  difference  with 
respect  to  the  tableau  with  no  failures  where  all  transitions 
are  possible.  These  empty  places  will  generate  zeroes  in 
the  transition  matrix  and  this  property  will  be  useful  later 
to  reduce  the  amount  of  computation  in  the  optimization  | 

process . 

Let  us  calculate  the  probability  transition  for  the 
entries  of  the  tableau: 
row  0:  initial  state  0 = (0  0) 


With  the  condition  CQ1  < CQ2/  there  will  be  a tran- 
sition from  state  0 to  state  3 if  there  are  no  failures. 
That  is, 

PQ  Q(t)  = (1-Pf)2  (4.7) 

With  only  one  failure  the  transition  will  be  to  states 
5 or  6 depending  on  the  failed  computer  and  the  pro- 
babilities will  be 

PQ  5(t)  = PQ  6(t)  = Pf(l-Pf)  (4.8) 

If  there  are  two  failures 


Except  for  the  above,  no  other  transitions  are  pos- 
sible . 

row  1 : initial  state  1 = (01) 

The  transitions  to  1,3  and  4 are  controled  by  the 
decision  variables  defined  in  chapters  II  and  III 
and  the  condition  that  no  failures  occur.  Denoting 
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by  P^(t)  the  probability  of  going  from  state  i to 
state  j for  the  case  of  two  computers  and  no  failures, 
the  following  transition  probabilities  can  be  written: 

pn=U-pf)2  = (l-Pf)2(l-a1©1) 


Pi3»ll-pf)2  p“  = U-P(>% 

P14-(i-p£)2  pJJ  = (l-Pt)2(l-c2>o.1e1 


(4.10) 


The  transition  to  state  2=(0  2)  (or  6= (2  0))could 
only  take  place  if  the  system  decided  to  go  to  state 
ls(0  1) (or  3*(1  0))  and  there  was  a failure. 

The  reason  is  that  there  is  already  a 0 in  the  state 
so  the  other  element  had  to  be  a 1.  Therefore 


P12=P4(1-Pf>  P11  = Pf(1'Pf) 


P16=Pf(1‘Pf»  P13  " £2al0l 


(4.11) 


The  transition  to  state  5=(1  2)  or  7= (2  1) 
can  happen  in  two  ways.  Either  the  system  decided 
to  go  to  state  3=(1  0)  or  1=(0  1)  respectively  and 
a failure  brought  computer  2 out  of  order  or,  the 
system  intended  to  go  to  state  4= ( 1 1)  and  the  same 
failure  happened  thus  we  have 


Pl^PfU-Pfl'p"  ♦ p")  - Pf(l-P£)a1e1 


(4.12) 


Pl7=Pf  (1-Pf)  + p“)  = Pf  (l-pf)  (l-e20tie1) 

Finally  the  transition  to  state  8= (2  2)  take  place 
when  there  are  two  failures,  no  matter  what  transition 
was  decided  upon,  hence 
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(4.13) 


row  2 : initial  state  2=(0  2) 

From  this  state  the  automatic  decision  is 
"bring  a copy  from  outside  to  computer  1".  If  there 
is  no  failure  in  computer  1 and  computer  2 remains 
out  of  order,  then  the  system  goes  to  state  5. 
Therefore 


P2  5=(1‘Pf) (1'pr)  (4.14) 

If,  on  the  other  hand,  computer  2 is  restored 
and  computer  1 does  not  fail  the  system  will  be  in 
state  3 in  the  next  instant  of  time  and  this  event 
will  occur  with  probability 

p 2 ( 1_pf ) pr  (4.15) 

Considering  now  the  case  when  computer  1 fails,  dif- 
ferent transitions  appear 


P2  6=PfPr 


(4.16) 


P2  8-pf<1-pr> 

row  3 and  row  4 are  obtained  in  the  same  way  as  row  1. 
row  5 : £ initial  state  5=(1  2)J  has  the  same  transition  pro- 
babilities as  row  2,  because  as  we  said  before,  the 
decision  of  going  from  2 to  5 is  automatic,  and 
then  the  possible  transitions  from  5 are  only  due  to 
failures  or  recoveries  in  the  computers (remember  the 
sequence  of  events  at  the  end  of  section  IV-3  that 


decided  a transition) . Thus 


row  6 


row  7 


row  8 


P5  5=P2  5=(1~Pf > (1-pr> 


P5  3=P2  3=(1’Pf)  Pr 


P5  6=P2  6=Pf  Pr 


P5  8=P2  8-Pf(1'Pf) 


^initial  state  6= (2  0)J  is  obtained  in  the  same  way 
as  row  2 but  interchanging  the  computers. 


P6  T(1-Pf)Pr 


P6  2=PfPr 


(4.17) 


P6  7=(1-Pf)(1-Pr) 


P6  8=Pf(1-Pr> 


^initial  state  7s (2  1)J  is  identical  to  row  6 for 
the  same  reason  than  row  5 was  identical  to  row  2. 


initial  state  8= (2  2) 

From  this  state  there  is  no  available  decisions 
and  the  only  solution  is  to  wait.  The  probabilities 
are : 


P =P 
8 0 r 


P8  2 P8  6 pr(1~Pr) 


(4.18) 


P8  8 (1_pr) 


The  vector  dynamic  equation  is 
Y(t+l)=P(t,u)Y(t) 


where  P(t,u)  is  the  transition  probability  matrix. 
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It  can  be  checked  that  the  transition  probability 
matrix  obtained  above,  is  a well  defined  stochastic  matrix  in 
the  sense  that  all  rows  add  up  to  one.  In  the  next  section 
we  will  see  that  these  results  easily  generalize  to  any 
value  of  NC  and  that  the  transition  probabilities  can  be 
obtained  easily  and  efficiently.  Then  in  the  next  chapter 
these  results  will  be  applied  to  a numerical  example. 

To  conclude  this  section  we  point  out  that  in  the  case 
of  a network  with  failures,  the  terms  due  to  updating 
traffic  are  not  only  function  of  the  state  to  go  but  also  of 
the  present  state  . The  reason  is  very  simple:  a computer 
that  is  not  working  cannot  generate  updating  traffic,  further- 
more if  the  state  is  such  that  no  copies  are  present  in  the 
system,  ((0  2)  for  instance)  no  updating  traffic  can  be 
generated  either.  A flow-chart  showing  how  to  compute  the 
updating  traffic  for  any  network  is  presented  in  the  next 
section . 

iv.  4.  Recursive  Equations  for  NO 3 and  a general  NC 
considering  node  Failures 

In  this  section  the  results  of  section  IV-3  will  be 
extended  to  any  NC  and  general  rules  showing  how  to  obtain  the 
per  unit-time-costs,  updating  traffic  costs,  and  transition 
probabilities  will  be  developed. 

First  the  case  NC=3  will  be  examined.  The  number  of 
states  is  M=33=27,  and  they  are  all  ternary  numbers  from 
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m=0  (0=(0  00))  to  m=26  (26= (2  22)).  Let  us  begin  deriving 


the  per  unit  time  and  terminal  costs.  We  have  the  follow- 
ing cases: 

a)  the  state  is  such  that  no  copies  are  present  and 
at  least  one  computer  is  working 


=min{C  .}  ^ C{set  of  working  computers| 


m ~ * "Oj 

b)  all  computers  are  out  of  work 


(4.20a) 


»M  * 0 


(4.20b) 


c)  general  case 

Am=  ^ Ci+  ^2  9^(t)  ^ {Cij}  (4.20c) 

i«I  j«J 

where  I is  the  set  of  working  computers  with  a copy,  and 
J is  the  set  of  working  computers  without  a copy. 

Some  of  these  costs  are  obtained  bellow  as  illustration 
(it  is  assumed  Cg^  < Cg2  < CQ3) 


0= 

(0 

0 

0) 

A0 

(t) 

II 

o 

o 

M 

4= 

(0 

1 

1) 

>4 

(t) 

= Cj+Cg+e^^min  (C21  ,C31) 

7= 

(0 

2 

1) 

> 

a7 

(t)  = 

=c3+b1c31 

18s 

(2 

0 

0) 

A18 

(t)  = 

=Cq2  <4-21’ 

24= 

(2 

2 

0) 

CN 

K 

(t)  = 

=C03 

26= 

(2 

2 

2) 

A26 

(t)  = 

= 0 

The  terminal  costs  are  obtained  in  the  same  way  except 
that  as  discussed  before,  there  will  be  a zero  in  the  place 
where  there  was  a Cg...  A flow-cha^t  showing  how  to  obtain 
terminal  and  per-unit  time  cost  for  any  value  NC  is  sketched 
in  Fig.  IV-3. 
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The  way  to  compute  the  cost  due  to  updating  traffic  for 
all  possible  transitions  in  a network  with  NC  computers  is 
shown  in  the  flow-chart  of  Fig.  IV,  2.  We  call  R(I,F)  to  the 
(I,F)th  element  of  the  updating  traffic  matrix. 


Data:  NC,  M,  0(i),  oi>  C... 

I = 1 ,M m 

F = 1,M m 

Write  I and  F in  base  3 with  NC  digits 
Ii'  Fi  iC|l»--NC|  are  the  digits 

IF2  to  the  number  of  2's  in  F (Final 

State) 

II  to  the  number  of  l's  in  I (Tnitia] 

State 

Fl  to  the  number  of  l's  in  F (Final 

State 

and  J 1(2)  to  set  of  indeces  3 1^  = 2 

(F(l)  to  set  of  indeces  3 Fj  = 1 


n-j) 

nJo 


F1=0  , or,  F2  = NC-1 

' . ' 

NO 


R ( I , F)  = £ Pk(t)0k(t) 

2 °kl 

k f I (2) 

ltF(l) 

1A 

Fig,  IV-2.  Flow-chart  of  updating  traffic  when  failures  in 
the  computer  are  considered 


DATA:  NC,M,  C^  C^,  CQi,  O^t) 


[0,T] 


i , j = 1 , . . NC  t L0' 

. f /Terminal  costs  A = 1 
1 (otherwise  A = 0 


Repeat  from  state  m=0  to  m=M-2 

Write  m in  base  3 with  NC  digits 
-*•  ith  digit  i = 1,..NC 


Let  us  call 
J = {j}  s.t.  itk  = 0 

I = {i}  s.t.  mi  = 1 

NJ  = number  of  elements  in  J 
NI  = number  of  elements  in  I 


X = C . 
m oj 


[NI  = 0 


V YES 

YES^ 

Let  j J s.t. 

■ 

r 

Coi  < Coi  ViCJ  1 * 3 

X = 0 
m 


x = y c,  + 
m .f  x i 


0S  (t)  min  (C  . 
3 iCl  13 


Vi  = 0 


_ /vector  A(t)  if  A = 

^Return  |Vector  v (T)  if  A = 


Fig.  IV-3.  Flow-Chart  showing  how  to  obtain  terminal 
and  per'-unit-time  costs 


■ *01011 
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In  fig.  IV-4  we  find  the  tableau  of  possible  transitions 
for  the  case  NC»3  with  symbols  as  defined  in  section  IV-3. 

The  basic  rules  for  the  formation  of  the  tableau  are  the 
same  as  for  the  case  N02; 

1)  No  transition  from  a state  comoonent  2 to  a state 
component  1 is  possible 

2)  If  there  are  only  0's  and  2's  in  the  state  vector 
the  only  possible  transition  is  to  bring  a copy 
to  the  computer  with  smallest 

Example  of  impossible  transitions 


(4.22) 


where  the  elements  causing  trouble  have  been  circled  out. 

To  obtain  the  general  rules  for  the  transition  proba- 
bilities let  us  begin  analyzing  some  examples. 

We  define,  as  before,  as  the  transition  probability 

from  i to  j in  a network  with  L computers  when  no  failures 
are  considered 

- The  transition  due  to  failures  or  recoveries  only 
are  obtained  with  the  same  basic  rules  as  for  the 
case  NC-2,  for  instance 


from 

0 

2 

2 

to 

0 2® 

from 

0 

0 

0 

to 

1 2® 

from 

0 

2 

2 

to 

®2  0 

from 

1 

2 

2 

to 

©2© 

0=(0  0 0) 
6=  ( 0 2 0) 
25= (2  2 1) 


9=  (1  0 0),  PQ  9=<1-Pf> 

17=  (1  2 2 ) , PQ  17=(l-Pf)  d-Pr)Pf  (4.23) 

2.(0  0 2>,P25  2-pJpf 


<D  0^OHMOHNOH(NOrlfNOHMOHMOH(NCHrJOHM 

4J  4-> 

<d  U(NOOOHHH(N(NMOOOHHH(NfSMOOOHHH(NM(N 

4J  0) 

W>HOOOOOOOOOHHHHHHHHH(NfMMfMMfMfNNN 


XXXXXXXXXXXXXXXXXXXXXXXXXXX 

XXX  X 

XX  XX  XX 

X XX  « « 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X X X X X X X 

X XXX 

X X X X X X 

X X * X X * 
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X XX  XX 

X XXX 

X X X X X X 
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X X X 8 

X X 8 X X 8 

X X X X X X 
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S B 0 0 

X XX 

8 8 8 X 

8 8 X 8 8 X 

X X X X X X 

XXX  X 

XX  XX  XX 

X XX  XXX 
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8 8 8 8 XX 

X XX  XX 

X 

Fig.  IV-4 . Tableau  of  possible  transitions  for  N C= 
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- The  transitions  from  states  with  no  inoperative 
computers  are  also  directed  by  the  same  basic  rules 
as  the  corresponding  transitions  for  NC«2,  see  section 
IV-3,  thus  from  state  1»(0  0 1)  to  state  10=  (1  0 1) 


pi  io-(1-pf)3  p”io 


(4.24) 


from  state  1=(0  0 1)  to  state  5=(0  1 2) 

P1  s-CWf)*pfCP”5  + PJ"| 

where  3 and  4 are  the  states  obtained  from  5 replac- 
ing the  2 by  0 and  1,  that  is 

5 = (0  1 2) 

3 = (0  10) 

4 = (0  1 1) 

- If  the  transition  is  to  a state  with  two  2’s  then  the 
number  of  ^ 's  contributing  to  the  probability 
is  larger  (3  or  4 depending  on  the  case) 
from  state  1*(0  0 1)  to  17  = (12  2) 

P1  17-;1-pf)2pf(P“9  + PJ”o  + PJ"2  + PW3>  (4-25) 

where  as  before  9,  10,  12  and  13  are  the  states 
obtained  from  17  replacing  the  2's  by  all  possible 
combinations  of  0's  and  l's,  that  is 

(1  0 0)  = 9 

from  17  = (12  2)  we  obtain  • (l  l o)  = 12 

(111)=  13 
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- If  the  transition  is  from  state  1=(0  0 1)  to  state 


24= (2  2 0)  then 

P «m-p  )2P  (P111  + P111  + P111  ) 
P1  24  U pf'  3*191  12J 


(4.26) 


where  now  we  only  have  three  terms  in  the  sum  because 
state  0= ( 0 0 0)  was  unallowed  for  the  case  of  no 
failures 

- The  probability  transitions  from  states  with  onlv  two 
computers  in  working  condition  will  be  formed  up 
from  terms  P*1^  . The  subscripts  i and  j will  be 
obtained  from  a modified  2-components  states  result- 
ing from  the  previous  one  after  the  elimination  of 
the  element  that  was  equal  to  2 in  the  initial  states 
and  applying  previous  rules.  That  is  from  state 
7={0  2 1)  the  modified  initial  state  will  be  (0  1)  = 1, 
therefore  a transition  from  7 to  11=  (1  0 2)  will  have 
the  following  probability 

P7  U'<1-pf)prpf(p"2  + P”3>  <4'271 

where  2 and  3 are  obtained  from  7 and  11  as  follows 


7=(O01) 
11= (1@2) 


(1  0)  = 2 
(1  1)  = 3 


The  rest  of  transition  probabilities  can  be  obtained  in 
the  same  way.  It  can  be  easily  checked  again  that  the  result- 
ing transition  matrix  has  all  the  properties  needed  by  a 
stochastic  matrix.  The  dynamic  equation  for  the  state  is, 
as  before, 

Y ( t+ 1 ) = P(t,u)  Y ( t ) 


These  results  generalize  to  a network  with  any  number 
of  computers.  It  will  nevertheless  be  convenient  to  sketch 


in  a flow-chart  the  ordered  set  of  steps  we  have  to  take 
to  obtain  the  element  of  the  transition  matrix.  To  simplify 
the  flow  chart,  let  us  suppose  that  we  have  the  following 


subroutines  available  (see  Appendix  C) 


BASE  3 (NC,N,NB3,N0,N1,N2) 
(3NC 


Write  N in  base  3 with  NC  components 
NB3 ( I ) is  the  Ith  component 


NO  = number 

of 

0 ' s 

in 

NB3 

> N)  j 

N1  = number 

of 

l's 

in 

NB3 

N2  = number 

of 

2 ' s 

in 

NB3 

BITS (N , NC , NB2 ) 


Write  N in  base  2 with  NC  digits 
NB2 ( I ) is  the  Ith  component 


ROWPRO  (NC,  v,  A ,0,  IT,  IND)  * 


Obtain  the  vth  row  of  the  transition 
probability  of  a network  with  NC 
components  and  no  failures,  when 
the  rates  are 


9(i)  i=l,NC  and 


the  decision  is  "go  to  state  X"  . 
This  result  is  obtained  with  the 
flow-chart  of  Fig.TII-3. 

JT  ( J ) is  the  Jth  element  of  this 


row 
IND  is 

IND 


an  ou 

, (i  1 

\0  o 


output  index  s, 
f someIt(J)=l 
otherwise 


(*) 


we  will  now  call 

N - the  row  (or  present  state)  being  computed 


(*)  Notice  that  if  this  happens  the  only  nonzero  element 
in  the  row  is  IT  ( X ) — 1 


M = the  number  of  states  M=  3 


NC  - the  number  of  computers  in  the  network  under  consideration 


the  number  of  states  for  the  auxiliar  subsystem  without 
failures 


R ( J , I ) - the  updating  traffic  cost  generated  in  a transition 

from  I to  J and  obtained  with  the  flow-chart  of  Fig.  IV-2 


the  Nth  column  of  the  matrix  R 


For  simplicity  of  representation  and  convenience  of 
computation  the  algebraic  representation  of  the  entries 
will  not  be  derived.  Instead  we  present  an  algorithm  to 


calculate  their  numerical  values  after  deciding  upon  the 


control  variables.  Having  the  transition  probabilities 


the  optimization  process  can  be  carried  out  directly  by 
moving  the  decision  to  all  its  possible  values,  performing 
the  product  with  the  matrix  (V  + R)  and  choosing  the  decision 


giving  the  smallest  value.  This  is  what  is  done  in  the  flow 


chart.  The  variable  \ will  control  the  admissible  decisions 


and  the  variable  L the  corresponding  element  within  the  row 


The  results  after  the  computation  will  be 


actual  optimal  decision 
from  state  N 


Go  to  state  LAMBOP 


scost  actual  cost  due  to  this  transition  resulting 

from  the  product  -*■  row  N x (V  + R^) 

Looking  carefully  at  the  flow-chart  it  can  be  seen  that 


the  first  thing  it  does  is  an  assignment  of  state  N to  one 


of  the  three  basic  types  of  states  it  considers.  In  this 
way  the  rest  of  the  computation  will  take  place  in  one  of 


ilSfettMfcdltaE-sJi.  * fe 


NC2> (WC-1) 


NF=NC 

NP=NC-1 I I MI=2 

J=NC+ 1 
NP=NC 


N2=NC-1  T^HLetJ  be  the  only 

component  s.t,  NB3(J)=2 


Let  J be  s . t 

C0J<C0k  vk*J 
I s . t . NB3(k)^2 
! NB3 [J)?2 


11=0, (LF-1 


MI  = 1 
NP=NC-1 


BA=0 (initialization 
-,NP 


LF=2 


Form  the  new  vector  W s.t 
r 0 if  lk=0 
”k  (2  if  Ik=l 


MM=MI , 2 


Form  the  vector  LB3  s.t. 
W (k)  if  k< J 
LB3  (k)  = m if  k=J 

W(k-l)  if  K J 


NZ=#  OF  O's  in  LB3{k)  NB3(k)=2 

NS=#  of  2 ' s in  LB3  (k)  NB3 (k) /2andk/J 


L=DECI ( 3 , LB2 ,NC ) 
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Branch  X corresponds  to  states  with  no  2 ' s amongtheir 
components 

Branch  Y corresponds  to  states  where  no  decision  is 
available 

Branch  Z to  the  rest  of  states.  This  branch  has  a sub- 
division depending  on  the  destination  state 
Z1  if  the  destination  state  is  such  that 
to  reach  it  we  have  only  one  possibility 
Z2  otherwise 


In  branches  X and  Z every  possible  transition  (index 
L)  from  N is  tested  and  only  the  transitions  that  give  a 
nonzero  probability  are  obtained.  In  branch  Y,  only  the 
nonzero  probability  transitions  are  generated.  Functioning 
in  this  way  the  algorithm  is  very  efficient  because  only 
the  nonzero  terms  (^30%  of  the  total)  are  obtained  and  there 
is  no  waisted  time  performing  zero  computation.  The  algorithm 
was  programmed  in  Fortran,  using  Assembler  for  BITS  Sub- 
routine (see  Appendix  C) 

Now  that  the  transition  matrix  and  optimal  decisions 
have  been  obtained  the  rest  of  the  process  is  identical  to 
the  case  with  no  failures.  (Fig.  III-4)  . The  whole  process 
is  represented  in  Fig.  IV-5  and  the  matrix  recursive  eq.  will 
be 

VMt)  = A (t) +P (t ,u* ) V*  (t+l)+P  (t,u*)BR(t)  (4.29) 

where  R(t)  is  now  an  M by  M matrix  and  ® means  that  we  make 
the  scalar  product  of  the  ith  row  of  P (. ,.)  with  the  ith  column 
of  R(.)  to  obtain  the  ith  component  of  a column  vector. 

Let  us  briefly  analyze  how  the  actual  dynamic  behaviour 
of  a network,  with  present  characteristics,  will  be,  when  the 


> 


ViCt  1,2 


e.(t) , pf/  Pr,  o ± (t 

{0,1,  ..NC}  tc:  1 , 2 


Compute  number  of  states 
M=3NC 

Compute  terminal  costs  (V(T)) 
(Flow-chart  Fig.  IV- 3 with  A=l) 


Steps  backward  in  time 


Compute  per  unit  time  cost  (A ( t ) ) 
(Flow-chart  Fig.  IV-3  with  A=0 


Rows  of  transition  matrix 


n = 1 , M 


Compute  row  n 
Obtain  optimal  decision 
and  minimum  cost  (SMt+1)  ) 
(Flow-chart  Fig.  IV-4 


Compute  new  costs-to-go 
V*  (t)  = A (t)  +S*  (t+1) 


Fig.  IV-6.  Optimization  Process  in  Network  with 


Failures 


. -.r-fr-ft. 


sequence  of  optimal  decisions  is  applied.  Let  us  suppose 
that  the  probability  of  recovery  for  a computer  that  is  not 
working  is  not  equal  to  1,  or  in  other  words  that  the  recovery 
is  not  instantaneous.  If  we  assume  that  the  process  begins 
with  all  the  computers  working,  the  system  will  begin  to  make 
transitions  among  states  with  no  2's  among  their  components. 
Once  a failure  takes  place  the  system  will  change  its 
"state  space"  to  states  that  have  a 2 in  the  position  of  the 
failure;  it  will  remain  making  transition  in  this  new  "state 
space"  until  one  of  two  possible  events  will  take  place: 
either  there  is  another  failure  (or  more) , or  the  failed 
computer  begins  to  work.  The  process  will  continue  in  this 
way.  We  see  therefore  that  the  whole  state  space  can  be 
divided  into  various  subspaces  such  that  the  system  will 
remain  most  of  its  time  making  transitions  in  those  subspaces 
and  eventually  will  move  from  one  subspace  to  another.  There 
will  be  as  many  subspaces  ''as  different  vectors  we  can  form 
with  NC  components  and  two  symbols  (one  for  2 and  the 
other  for  1 or  0) . All  the  vectors  with  same  frame  of  2 
components  (and  at  least  one  1)  will  belong  to  the  same  sub- 
space . 

For  NC=3  some  of  the  subspaces  will  be: 

0 0 1-1 

010-3  [0  12  [0  21 

S.  011-4  S 1 0 2 S,  1 2 0 (4.30) 

100-9  ^ 1 1 1 2 J |l  2 1 

1 0 1 - 10 
1 1 0 - 12 


This  particular  behavior  and  subspaces  division  can 
also  be  illustrated  if  the  transition  tableau  of  Fig.  IV-4 


I 

1 


I 


: 


is  written  with  a different  state  order  as  in  Fig.  IV-6 
is  shown.  The  system  will  be  most  of  the  time  within  the 
marked  regions  of  the  tableau  and  eventually  will  move  from 
one  region  to  another.  The  states  with  NC-1  elements  equal 
to  2 (as(0  2 2)  = 8)  may  be  thoughts  as  degenerated  subspaces 
or  just  subspaces  with  only  one  component. 
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CHAPTER  V 

NUMERICAL  APPLICATIONS  AND  OTHER  ANALYTICAL  RESULTS 

V. 1 Time  varying  rates, no  failures,  no  updating  traffic. 

Let  us  analyze  the  case  of  a network  with  two  computers 
sharinq  a file  according  to  time  varying  rates.  We  will 
apply  the  model  of  chapter  III  and  we  want  to  know  the  dynamic 
evolution  of  the  states  (i.e.  allocation  of  the  file)  in 
order  to  minimize  the  total  expected  cost.  We  also  want 
to  study  the  evolution  of  the  state  dynamics  and  total  cost 
as  the  storage  cost  varies  from  0 to  the  value  of  the  trans- 
mission cost.  The  transmission  costs  will  all  be  taken  to 
be  equal  to  1.  The  problem  will  be  solved  for  storage  costs 
equal  to  0,  0.25,  0.5,  0.75,  1.  The  system  will  operate  for 
a period  of  20  time  units,  |l,20] . The  rates  are  represented 
in  Fig.  V.l,  and  the  results  inTable  V.l.  In  the  columns 
called  "evolution  of  states"  we  write  the  optimal  decision 
("go  to  state...")  for  every  possible  initial  state  and  every 
instant  of  time. 

Examining  the  table  we  can  see  that  for  the  case  of 
storage  cost  equal  to  zero  the  optimal  decision  is  to  always 
keep  a copy  of  the  file  in  each  computer.  The  optimum  initial 
state  is  state  3 and  the  optimal  decision  being  at  state  3 
is  always  "remain  in  3".  This  is  the  logical  result  because 
there  is  no  payoff  for  keeping  a copy  in  any  computer.  At 
tel9  we  will  leave  the  system  at  state  2;  this  decision  is 


■■■«* 
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RATES  C^-  1 Vi,3 


COMPUTER 
1 2 

0.8 

0.0 

0.8 

0.4 

0.8 

0.8 

0.8 

0.8 

0.0 

0.8 

OPTIMUM  INITIAL 
STATE 


MINIMUM  TOTAL 
EXPECTED  COST 


OPTIMAL  DECISIONS 
EVOLUTION  OP  STATES 


222  222  222 
333  333  222 
333  333  333 
133  133  133 
113  113  HI 
313  313  HI 
313  313  HI 
333  222  222 
333  333  222 
133  133  133 
133  133  111 
133  133  111 
133  111  111 
333  111  HI 
333  333  333 
333  333  333 
333  333  333 
313  313  111 
113  113  111 


3 3 
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taken  because  we  know  that  at  t*20  there  will  be  no  request 
coining  from  computer  1.  Nevertheless  we  could,  in  any  case, 
remain  in  state  3 without  increasing  the  cost,  so  both 
decisions  give  the  same  cost. 

As  we  increase  the  storage  cost  the  number  of  optimal 
decisions  that  are  different  from  "go  to  state  3"  is  larger 
and  finally  when  the  storage  cost  is  equal  to  1 the  optimal 

decision  will  always  be  go  to  states  1 and  2 only,  that  is, 

keeping  only  one  copy  in  the  system  at  any  time. 

Looking  now  at  the  columns  of  Table  V.l  we  see  that  the 
optimal  cost  increases  in  a nonlinear  fashion  as  the  storage 
cost  increases. 

For  comparison  we  also  consider  a static  analysis  with 
the  corresponding  average  rates: 

20 

e^t)  =0.41  Vt-.e[l,2o]  (5.1a) 

t=l 
20 

e2(t)  = 0.40  Vtc[i,2o]  (5.2b) 

t=l 


•;v“) = io 


av.  , _ 1 
e2  (t)  “ JO 


We  have 


storage  cost 

optimal  allocation 

total  cost 

0 

• 

two  copies 

0 

0.25 

two  copies 

10 

0.50 

one 

copy  at  comp. 

1 

18 

0.75 

one 

copy  at  comp. 

1 

23 

1.0 

one 

copy  at  comp. 

1 

28 

* 
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The  corresponding  total  costs  are  larger  here,  but  not 
significantly.  This  is  because  we  are  considering  a very 
short  period  of  operation. 

The  curves  describing  the  evolution  of  the  total  cost 
as  the  storage  cost  varies  are  represented  in  Pig.  V.2  for  the 
state  and  dynamic  cases.  We  also  represent,  in  the  same 
figure,  the  two  curves  corresponding  to  the  rates  of  Fig. 

V . 3 and  the  curves  corresponding  to  a network  with  3 compu- 
ters (NC=3)  with  rates: 


vte(I,2o] 


©1  (t)  =©i  (t)|cage  i 

02(t)=e2(t>|case  1 
93(t)=e2(t)|CaSe  2 


We  can  see  that  in  all  cases,  they  have  a similar  shape.  In 
particular  case  1 and  case  2 have  the  same  static  curve. 

This  is  due  to  the  fact  that  in  both  cases  ©*v  > ©^  , 

©2  (case  1)  = ©2  (case  2)  and  the  optimal  allocations  are 
the  same  for  every  value  of  the  storage  cost,  as  can  be 
easily  checked. 


ij 


L i 

1 


V . 2 Constant  Rates.  Updating  Traffic.  No  Failures . 

In  a practical  case  it  could  be  very  difficult  to  specify 
the  rates  as  detailed  as  a time  variant  function,  even  with 
piecewise  constant  shape.  It  seems  more  reasonable  to  model 
the  rates  as  constant  functions  over  a period  of  time. 
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An  intermediate  case  will  be  to  obtain  the  rates  as  piecewise 
constant  functions  with  long  steps.  In  this  case  a quasi- 
dynamic analysis  applying  the  optimization  procedure  to  every 
long  step  separately  can  be  considered.  In  any  case  it  is 
important  to  analyze  carefully  the  behavior  of  the  system 
with  constant  rates  of  demand. 

Let  us  suppose  that  we  have  a network  with  three  com- 
puters (NC=3) , with  demand  rates 

©1(t)=0.8  ©2 (t) =0 . 6 ©3 (t) =0 . 4 Vt  1,8 

and  an  operating  period  equal  to  eight  time  units  (T»8) . As 
before  we  consider  the  transmission  cost  equal  to  one  for 
every  possible  transmission 

Ci;j=  1 Vi,j«{l,2,3} 

The  storage  costs  and  updating  ratios  are  the  same  for 
all  computers 

C.  = C 

s Vi C {1,2,3} 

= P 

and  we  want  to  analyze  the  system  for  the  values 

C = 0,  0.25,  0,5,  0.75,  1 
s 

p = 0,  0.25,  0.5,  0.75,  1 

The  results  for  p = 0.25  and  Cg  = 0,  0.25,  0.50  are 
shown  in  Tables  V-2.  We  will  represent  later  the  evolution 
of  the  total  cost  as  the  storage  cost  varies  taking  p as  a 
parameter.  Again  we  will  make  linear  interpolation  between 
exact  points. 


©1=0 . 8 


©2=0.6 


C =0.0 
s 


p=0 . 25 


Decision 


0.00 


6 6 6 6 6 6 6 
6 6 6 6 6 6 6 
6 6 6 6 6 6 6 
6 6 6 6 6 6 6 
6 6 6 6 6 6 6 
6 6 6 6 6 6 6 


1.65 

1.45 

1.30 

1.25 

0.90 

■clU iH'KT.'M'JI  :l 

4.67 

7 4.42' 

4.27 

4.18 

4.03 

3.80 

6.12 

, 5.87 

5.72 

5.63 

5.48 

5.25 

7.57 

, 7.32 

7.17 

7.08 

6.93 

6.70 

"9.03" 

8.77 

8.62 

8.53 

8.38 

8.15 

10.48 

10.22 

10.07 

9.98 

9.83 

9.60 

6.55 


9.45 


6 6 6 6 6 6 6l  11.93  11.67  11.52  11.43  11.28  11.05110.90 


Decision 


4 4 4 4 4 4 4 
4 4 4 4 4 4 4 
4 4 4 4 4 4 4 
4 4 4 4 4 4 4 
4 4 4 4 4 4 4 
4 4 4 4 4 4 4 
4 4 4 4 4 4 4 


p=0 . 25 


3.75 
‘5.52 

7.27  7.01  7.22  6.75  6.85  6.65  6.75 

9.02  8.76  8.97  8.50  8.60  8.40  8.50 

10.77  10.51  10.72  10.25  10.35  10.15  10.25 

12.52  12.26  12.47  12.00  12.10  11.90  12.00 

14.27  14.01  14.22  13.75  13.85  |13.65  13.75 


1-  50 
5.26 
7.01 
8.76 


_3._69_|  3.25  3.35 

5.47  5.00  5.10 

7.22  6.75  6.85 

8.97  8.50  8.60 


3.1 
4.9 
6.6 
8.40 


1.50 


3.25 

5.00 

6.75 

8.50 


Table  V . 2 . 


101 


The  first  important  characteristic  that  appears  looking 
at  these  tables  is  that  there  are  no  transients  in  the  decisions. 
This  is  not  a general  statement  that  can  be  applied  to  any 
network  and  parameters  but  will  motivate  us  for  a deeper 
analysis  of  the  facts  that  are  taking  place  in  the  optimi- 
zation algorithm  with  the  assumptions  of  this  section.  With 
these  assumptions  we  observe: 

a)  The  vector  A(t)  is  time  invariant 

A (t)  = A Vt  C [l,T] 

b)  The  transition  matrix  P(tfu*)  corresponding  to 
optimal  decision  is  also  time  invariant 
P(t,u*)  = P(u*) 

c)  The  updating  ratio  vector  (that  had  to  be  added  to 
cost-to-go  vectors)  is  also  constant.  Let  us  call 
this  vector  R 

d)  The  terminal  cost  vector  is  equal  to  the  vector  A 
V (T)  = A (T)  = A 

Therefore  we  can  write  the  recursive  equation  for  the 
first  iteration 

V(T-l)  = A+P(u*)  (A+R)  (5.2) 

We  want  to  show  first  that  with  these  conditions,  the 
system  exhibits  a trapping  state. 

Let  us  call  A+R  = ft1  where  S21  = ^ 


and  let  us  suppose  that  w . is  the  smallest  component  of 
vector 


1 

j 


(5.3) 


Vj  ¥ i j«{l,2,..M} 
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For  the  system  can  remain  in  the  same  state,  if  this  is 
the  decision,  with  probability  one,  the  ith  row  (correspond- 
ing to  state  i)  of  P(u*)  will  be  then 

i 

ith  row  — *>  (000 010 0)  (5.4) 

furthermore  if  we  represent  by  K the  set  of  states  having 
the  same  1 components  as  state  i and  at  least  one  1 com- 
ponent more  (we  will  call  this  set  the  optimum  set)  we  have 

i 

kth  row  of  P(u*)  « (00--010--0)  Vk*K  (5.5) 


if  we  call  now 

P(uJ)n!  = **  where  <J>  * = (<j>* , <J>2  , • • 4>^)  T 
we  have 

^l  = a)i  v leK  or  l=i 


(5.6) 


(5.7) 


^m  > ^1  Vme  {1,2,  . .M},m/1 
Writing  now  the  second  backward  step  in  the  recursive  equation 
we  have 

V (T-2)  = A+P(u*)  [V (T-l)  + Rj  = 

= A+P(u*)  [ A+  ♦*+  R]  = A + P(u*)n2  (5.8) 

where 

ft2  = A+4> 1 + R =a’  + 1 (5.9) 

from  (5.3)  and  (5.7)  we  have  again 

u>^  c go2  Vj  ? i j < { 1 , 2 , . .M]  (5.10) 

and  hence 

i 

kth  row  of  P(u*)  = (0  0 - - 0 1 0 - - 0)  Vk«K  or  k=i 

(5.11) 

i.e.  the  optimal  decision  from  state  i is  again  "remain  in 
state  i";  and  this  decision  can  be  implemented  with  probabi- 


lity  one.  From  this  result  we  obtain,  as  before 

*!  « „}  VI  ek  or  1 = i (5.12) 

e*  > Vme  (1,2, . .M  } m j 1 (5.13) 

m J. 


where 


p (u*)  n2=  $2=  ($2,$2, . 2)T 


(5.14) 


This  characteristic  or  behavior  will  be  repeated  through  the 
remaining  steps.  Therefore  state  i will  be  an  absorbing  or 
trapping  state,  according  to  the  most  frequent  nomination 
for  these  kinds  of  states  in  the  literature  (see  refs. 

[18}  and  [l9]).  This  means  that  once  the  system  visits  i 
it  remains  there  forever.  Furthermore  if  the  process  is 
long  enough  we  may  expect  that  the  optimal  initial  (at  t»l) 
allocation  will  correspond  to  state  i or  any  state  ktK  and 
thus  the  system  will  fall  off  in  state  i since  the  very 
beginning . 

The  former  property  suggests  a very  efficient  and 
quasi-optimal  procedure  to  analyze  a system  with  the  above 
characteristics.  We  can  describe  this  procedure  as  follows: 

A quasi-optimal  steady-state  (Vt  Ql,T])  decision  for  a 
system  with  constant  parameters  and  no  failures  will  be 
to  allocate  the  file  according  to  the  description  of  state  i 
with  i obtained  from  the  condition 


(5.15) 


where 


This  quasi-optimal  decision  role  can  be  considerably 
far  from  the  true  optimum  if  the  terminal  costs  were  very 
different  in  value  from  the  vector  A,  and  if  the  operating 
period  were  too  short  as  to  be  able  to  disguise  the  influence 
of  the  terminal  costs.  We  can  see  that  this  decision  is  in 
fact  the  decision  we  had  arrived  to  if  the  analysis  had  been 
made  under  the  so  called  static  approximation. 

Although  the  former  decision  could  be  enough  for  our 
purposes,  since  the  concept  of  trapping  state  is  conclusive 
in  the  performance  of  this  kind  of  process,  it  does  not  clarify 
completely  the  lack  of  transients  in  the  iterations.  An 
intuitive  and  heuristic  idea  in  this  direction  may  come 
thinking  that  the  optimization  process  will  be  very  much 
biased  by  the  states  belonging  to  the  set  K defined  above. 

This  bias  will  be  in  the  sense  that  any  state  not  in  the  set 
K will  try  to  move  toward  some  of  the  states  belonging  to  K 
in  order  to  reduce  the  cost.  (In  some  sense  this  set  K 
could  be  interpretated  as  a "stopping  set"  using  De  Leve 
terminology  [2l] ) . This  behavior  is  represented  graphically 
in  Fig . V . 4 . 

In  particular,  for  the  case  p=0.25,  Cs=0.25  of  the 
example  we  are  considering, Table  V.2,  the  diagram  of  optimal 
transitions  is  represented  in  Fig,  V.5. 


Let  us  go  back  to  Table  V.2  and  look  at  the  minimum 
cost  and  optimal  initial  states  . We  have  worked  out  the 
minimum  total  cost  for  every  instant  of  time,  and  the  optimal 
initial  states  assuming  that  the  process  might  begin  at  some 
time,  not  necessarily  at  time  1.  We  can  see  several  interest 
ing  results  looking  at  those  marked  numbers  of  the  table, 
namely:  For  a given  parameter  values: 

a)  The  minimum  total  cost  up  to  any  instant  of  time 
always  corresponds  to  the  same  initial  state. 

b)  Consequently  the  optimum  initial  state  is  always  the 
same  for  every  time. 

c)  The  increasing  of  the  minimum  total  cost  is  linear 
with  time. 

d)  The  optimum  initial  state  does  not  necessarily 
coincide  with  the  omnipresent  optimum  decision. 

Do  these  conclusions  generalize  to  any  system  with  constant 

parameters?  Let  us  investigate  this  question. 

From  previous  discussions  it  is  clear  that  the  optimum 
initial  state  will  belong  to  the  optimum  set  of  states,  the 
set K.  Remember  that  if  the  terminal  costs  were  dif  - rent  from 
the  p.u.t.  costs  we  had  to  wait  until  the  steady  state  were 
reached  in  order  to  make  this  statement;  otherwise  the  opti- 
mum initial  state  does  not  need  to  be  in  the  optimum  set. 

For  the  states  belonging  tq  the  set  K we  can  write  the  recur- 
sive equation  as  follows: 

V (t)  = A.  + V (t+1)  + r Vk«K  (5.17) 

i (trapping  state)eK 
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and  for  the  next  step 

Vk(t-1)  = + r±  (5.18) 

therefore 

Vk(t-1)-Vk(t)  = V^tJ-V^t+l)  = Xi+ri-u)i  (5.19) 

but 

vk (T)  = Xk  (5.20) 

so  we  can  write 

vk  (t)  = Ak  + (T-t)  u>i  Vke  K (5.21) 

This  important  result  will  answer  many  of  our  questions. 

In  fact 

\ . < A.  v k,j£K  =£  V.(t)  ^ V.  (t)  Vk , j e K 

] k k^j  3 k k^j 

vte  [i,t] 

Hence  the  minimum  total  cost  will  always  correspond  to  the 
same  state.no  matter  how  long  the  operating  period  is. 
Obviously  that  state  will  be  the  optimum  initial  state  for 
any  instant  of  time. 

Furthermore,  the  minimum  total  cost  increases  linearly 
with  time  as  it  was  expected. 

So  far  we  have  proved  that  facts  a) , b)  and  c)  are 
valid  for  any  system  with  constant  parameters.  Concerning 
fact  d)  it  is  clear  that  the  state  with  minimum  per  unit 
time  cost  among  the  optimum  set  need  not  be  the  trapping 
state,  but  if  the  optimum  set  only  has  one  element  (as  in 
the  case  Cg= 0,  p=0. 25)  then  it  is  clear  that  this  element 
has  to  be  the  trapping  state. 
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We  can  see  that  the  problem  with  constant  parameters  and 
terminal  costs  equal  A falls  in  the  section  of  problems  where 
the  optimum  decision  is  the  decision  that  minimizes  the 
immediate  cost. 

Before  leaving  Table  V.2  it  can  be  useful  to  make  a few 
more  comments.  For  instance  we  will  not  always  find  a 
unique  optimal  decision,  no  matter  which  state  or  time  we 
are,  as  the  one  shown  in  that  table;  as  an  example  the  case 
Cg=l,  p=0.5,  below  shows  a transition  from  3 to  2 as  inter- 
mediate step  to  arrive  to  state  4. 

Table  V.3 


rimel 


8 


eI=o.30 
Decision  I 


©2=0.60 


©3=0.40 


p—  0.5  Cs=l 


2.001  2.60  2.40  3.00 


5. 

02 

4, 

.76 

5, 

IjTjl 

7.54 

*~T. 

,27“| 

8, 

.16 

10. 

05 

9. 

.77 

10. 

.67 

12. 

55 

12. 

.27 

13. 

.17 

15. 

05 

14. 

,77 

15, 

.67 

17. 

55 

17. 

,27 

18. 

,17 

20. 

05 

19. 

77 

20. 

.67 

5.10 

7.60 

10.10 


4.90 
7.40 

9.90  10.50 


20.10  19.90  20.50 


This  is  due  to  the  fact  tnat  a decision  of  moving 
directly  from  3(0  1 1)  to  4(1  0 0)  has  some  intrinsic  risk 
of  remaining  in  3 if  no  request  is  made  from  computer  1. 

It  turns  out  that  in  this  case  a large  price  has  to  be  paid 
if  the  system  remains  at  3.  On  the  other  hand  a transition 


to  state  2 can  be  done  with  probability  one  and  carries  with 
it  a smaller  cost.  It  can  be  expected  that  in  a more  general 
case  with  more  computers  and  consequently  a larger  variety 
of  states  this  event  will  appear  more  frequently. 

An  important  observation  is  to  note  that  clk  can  be  iden- 
tified with  the  steady  state  gain  of  the  process  as  defined 
by  Howard  [20^  . Equation  (5.21)  for  optimum  cost-to-co  will 
justify  our  assertion  if  we  can  show  that  the  gain  is  unique 
even  under  optimal  policies  that  make  the  Markov  process 
non  ergodic. 

The  only  type  of  policies  that  can  make  the  system  non 
ergodic  (in  the  sense  defined  by  Howard)  will  be  those  con- 
taining at  least  two  persistent  decision  of  the  form  "remain 
in  ij"  where  now  each  i^  will  be  a trapping  state.  We  will 
show  that  if  1 and  m are  two  of  these  trapping  states  then 

1 m 

To  see  that  let  us  call  K(l)  and  K(m)  to  the  corres- 
ponding optimal  sets.  Notice  that 

K ( 1 ) n K(m)  f <J>  ($=  empty  set) 

because  K(l)  K (m)  will  contain  at  least  the  element 

M= ( 1 1 . . . A ) . Therefore  if 
r e K (1)  n K (m) 

when 

vr(t)  = >r(t)  + (T-t)  w = 


A (t)  + (T-t)  k> 


Ill 


and  hence 

1 m 

We  conclude  that  the  gain  is  always  unique,  even  in  the 
case  that  the  policy  would  make  the  system  non  cryodic. 

The  discontinuous  staircase  line  in  Table  V.2  represents 
the  beginning  of  a constant  Increase  in  costs  for  every 
state  of  the  process. 

Let  us  analyze  now  the  evolution  of  the  total  cost 
versus  storage  cost  taking  the  updating  ratio  as  a parameter. 

The  curves  are  shown  in  Fig.  V.6. 

First  of  all  we  see  that  the  shape  of  the  curves  is 
similar  to  the  one  obtained  in  section  V.l  for  the  time 
varying  case  without  updating  traffic.  The  curves  present 
a larger  curvature  for  small  values  of  Cg  and  p and  then  the 
behavior  is  almost  linear.  In  Fig.  V.7  we  represent  the 
total  minimum  cost  versus  p taking  now  Cg  as  a parameter. 

We  can  see  that  both  sets  of  curves  are  quite  close  in  shape 
but  the  curves  of  total  cost  versus  p have  a deeper  slope 
and  after  the  curvature  section  they  are  completely  linear. 

We  could  obtain  a better  understanding  of  this  curve  if  we 
divide  the  whole  quadrant  in  three  sections  corresponding 
each  section  to  trapping  states  with  the  same  number  of 
copies.  In  this  case  we  will  have  a section  for  three 
copies,  other  section  for  trapping  state  with  two  copies 
and  the  third  one  for  only  one  copy. 


♦ 

[ iillii'  ■ : i 

r. . . . 

i . . . . 

— 

• . «- 

. • i • 

1 , - 

, . . -i- 

- -f-i- 

. . T I 

« i - 
. • 1- 

--  -t  ♦ • 

1 i : : 

i • • » 

; . 

. t 

i . • • 

. ! 1 4 

. . . 1. 

4 • • 

■ * r 

dhi. 

Tota 


114 


'■f 

r-  -4 


At  each  of  the  Doints  of  the  curves,  corresponding  to 
the  cases  studied  in  the  present  example,  we  have  encircled 
the  trapping  state  corresponding  to  the  system  with  the  par- 
ticular parameter  assigned  to  the  point.  We  can  see  now 
that  linear  (or  almost  linear)  part  of  the  curves  corres- 
ponds to  points  with  the  same  trapping  state  while  the  rounded 
parts  are  due  to  changes  in  this  state. 

We  could  now  ask  the  following  question  regarding  the 

trapping  states.  Why  the  trapping  state,  having  two  copies, 

is  always  state  6(110),  and  why  the  trapping  state  having 

one  copy  is  always  4(100)?  The  reason  is  very  simple. 

Remember  that  our  rates  were  such  that 

©3  = 0.4  < ©9  = 0.6  < = 0.8  (5.26) 

therefore,  other  parameters  being  equal,  the  system  will 
try  to  keep  copies  in  the  computer  with  higher  request 
rate  and  there  is  no  reason  why  it  should  behave  in  a different 
way.  This  fact  is  easily  generalizable  and  we  can  say  that 
if  we  enumerate  the  computers  according  to  the  sequence  of 
decreasing  values  of  the  rates 

9.  > 90  > 9,  > 9.  > > 9m_  > 0 (5.27) 

1-  2—3—  4—  — NC 


the  trapping  state  will  always  have  its  "1"  components  con- 
centrated in  the  left  of  its  vector  representation. 

Let  us  now  analyze  for  what  values  of  Cs  and  p wi) 1 the 
trapping  state  change.  Concentrating  first  on  the  case  p=0 
we  can  see  that  the  change  from  trapping  state=7  to  trapping 

state=6  will  be  at  some  point  in  between  C -0.25  and  C =0.5. 

s s 

We  claim  that  the  frontier  will  be  narked  by  the  point  at 


, 
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which  the  system,  being  in  state  6,  is  indifferent  tc  a 
transition  to  state  7 to  as  opposed  to  remaining  in  state 
6.  A small  perturbation  from  this  equilibrium  point  will 
require  the  system  to  go  to  7 (if  the  perturbation  is  to 
the  left)  or  to  remain  in  6 (if  the  perturbation  is  to  the 
right)  . 

The  reason  is  again  very  simple.  State  6 cannot  be 
a trapping  state  if  being  in  this  state,  the  system  decides 
to  move  to  some  other  state.  On  the  othet  hand,  once  the 
system  decides  to  remain  in  6,  this  state,  considering 
all  the  decisions  above,  becomes  a trapping  state. 

Let  us  find  this  equilibrium  point.  Consider  that  the 
system  is  at  state  6 = (1  1 0)  at  time  t-1,  then  we  have: 
Cost  due  to  a transition  from  6 to  6 

* (6.6)  = Vg (t)  + rg  (5.28) 


Cost  due  to  a transition  from  6 to  7 s (111) 

*(6.7)  = e3[v7(t)+r7]  + (l-e3)  |vg(t)+r6J  (5.29) 

Equilibrium  point  (assuming  0^  / 0) 

$ (6.6)  = *(6.7)  V?(t)  + r?  = Vg(t)  + rg  (5.30) 

but  we  know  from  (5.21)  that 


V?(t)  = X?  + (T-t)  uk 
V6(t)  = Xg  + (T-t)  uk 

therefore  the  equilibrium  point  will  be  at 
A?  + r?  = X6  + r6 

(notice  that  in  fact  Xg  + rg  = X7  f r7  = 


(5.31) 


(5.32) 


u>i) 


r?=2p (e1+e2+e3) 


(5.33) 


A6  = 2Cs+03  r6=p(0l+92+2e3) 

and  this  substitution  yields 

X7  + r7  = X6  + r6  Cs  + P(01  + 02}  = 03  (5-34) 

So  we  have  obtained  a linear  relation  between  P and  Cg 
that  describes  the  equilibrium  point.  For  p=0  the  equilibrium 
is  at 

C = ©,  = 0.4  (5.35) 

s 3 

For  P = 0.25  the  equilibrium  is  at 


C = e,  - 0.25  (e.+e,)  = 0.4  - 0.35  = 0.05  (5.36) 

S 3 i Z 

The  intersection  of  the  equilibrium  line  with  the  horizon 
tal  axis  will  correspond  to  the  cost  due  to  the  equilibrium 
point  with  Cg  = 0 


Cs  = 0 =£  P * e3/ei+e2  = °-29 


(5.37) 


V?(l) 


= (8-l)ui  = 7 ( 1 ?+r? ) =7.20 


(5.33) 


C 


= 0 


s 

p =0 . 29 


We  can  see  that  for  the  given  rates  the  equilibrium 
above  is  possible  only  if  p <^0.29 

With  the  same  reasons  as  before  we  claim  now  that  the 
equilibrium  between  trapping  state  6(110)  and  trapping  state 
4(100)  will  be  defined  by  the  points  at  which  the  system, 
being  in  state  4,  is  indifferent  to  a movement  toward  state 
6 or  staying  in  state  4. 
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The  new  equilibrium  line  will  be  defined  by 
♦ (4,4)  = ♦ (4,6) 


(5.39) 


where 

♦ (4,4)  = V4(t)  + r4 

<M4,6)  = e2[v6(t)  + vj  + d-e2)  [vA(t)  +rv4] 

then 

♦ (4,4)  = ♦(4,6)  S=5  V4(t)  + r4  = Vg(t)  + rfi 

but 

v4(t)  = x4  + (T_t)  wi 
V6(t)  = Ag  + (T-t)  ok 

with 

A4  * Cs  + ®2  + 93  r4  ' + (e2  + ®3> 

*6  ’ 2Cs  + ®3  r6  = 4,(91+S2+2e3) 

hence  the  equilibrium  line  is  defined  by 

X4  + r4  = X6  + r6  ” Cs  + = 9 2 


(5.40) 


(5.41) 


(5.42) 


(5.43) 


(5.44) 


therefore 

o = 0 = ^>  C =9o=0 . 6 

s 2 

P = 0.25  ==3  C =0. 3 (5.45) 

s 

P = 0.5  =£  Cs  = 0 

Now  after  all  these  properties  have  been  described  we 
are  ready  to  provide  more  information  about  linearity  and 
curvature  of  the  graphs  of  Fig.  V.6  and  V.7.  But  before  doing 
that  let  us  generalize  the  results  obtained  above. 
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We  claim  that  in  a network  with  constant  parameters 
and  NC  computers  numbered  in  such  a way  that 


e!-  ^2  — 03  - • ' ’ i 0NC-1  ± 9NC 


(5.46) 


the  equilibrium  line  separating  the  trapping  state  with  NC 
copies  from  the  trapping  state  with  NC-1  copies  will  be 
defined  bv  the  relation 


♦ (M-l,  M— 1)  = (M~1,M) 


(5.47) 


where 

M=2NC-1 

Similarly  the  equilibrium  line  separating  the  trapping 
state  with  NC-1  copies  from  the  trapping  state  with  NC-2 
copies  will  be  defined  by  the  relation 


<>(n,u)  = <Mu,  M-l) 

where  u is  described  by  the  vector  (1,1,1,..  1,0,0) 


(5.48) 


l . e . p 


= ^ 2n  = M-l -2  = M-; 


(5.49) 


In  general  the  equilibrium  line  separating  trapping 
states  with  n and  n-1  number  of  copies  will  be  described 
by  the  relation 

t(v,v)  = <Mv,v  + 2) 


where 


v = (3,  1,  ..  1,  0,  ..  0)  , i,e.v  = 2r 


NC-n+1 


m=NC-n 


(5.50: 


We  can  arrive  to  an  explicit  relation  of  this  equilibrii 
line  as  a function  of  the  parameter  of  the  system  if  we 


v 


) by  its  expression  in  function  of  those  para 


meters.  In  fact  if  we  are  in  state  v at  time  t-1  we  have 


but  remembering  that 


we  will  have  equilibrium  if  and  only  if 


but  from  (3.6)  and  (4.1)  we  know  that 


m=n 


m=n+l 


therefore  we  will  have  equilibrium  if  and  only  if 
NC  NC  NC 


(n-l)  Cs*  J2 


e + 

m 


m=n 


NC 


= nC  + 
s 


© + p 

m 


° ['-2>  E ®ni  + E Sm]  * 

m=l  m=n 

NC  NC 

[(n-1>  E em  + E 


m=n+l 


m=l  m=n+l 


(5.59) 


that  is  iff 


NC 


© = C + P 

n s 


e 


m 


m=l 

m^n 


(5.60) 


Let  us  now  calculate  the  total  minimum  expected  cost.  Clearly 
this  cost  will  be  given  by 

Vjd)  = Aj  + (T-l)  u)i  j,  i C k (5.61) 

where 


A.  < A,  Vk^j  k e K 
J K 


u> . = A + r =A.„+r^_ 
l v v v+2  v+2 


i = v+2  or  i = v in  the  equilibrium  line  because 

(5.62) 

and  i = v+2  to  the  left  of  this  line  whereas  i = v to  the 
right.  The  minimum  cost  in  the  left  side  of  the  equilibrium 
line  (without  trespas.sing  the  area  where  v+2  is  the  trapping 
state)  will  be  then 


= A , + A _ + r 

j v+2  v+2 


ie'< 


1 


(5.63) 


V*  (1) 
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whereas  in  the  right  side  and  before  crossing  another 
equilibrium  line  the  cost  will  be 

Vr(l)  = X . +A  + r j Kr  (5.64) 

] 3 v v J 

It  is  important  to  notice  that  j does  not  have  to  be 
the  same  in  both  expressions  because  when  we  move  the  point 
up  and  down  on  the  curves,  we  are  changing  Cg  or  p and  j 
will  be  a function  of  Cg.  Furthermore  the  set  K (remember 
j K)  will  be  increased  by  one  element, the  new  trapping  state 
every  time  we  cross  an  equilibrium  line. 

Let  us  express  these  two  costs  in  terms  of  the  para- 
meters of  the  system. 


NC 


V1(l) 


= min  < tC  + j 0 > 

nor<NC  ( S ^ m J 


+ nC  + A + PB 


(5.65) 


m=i+l 


Vr(l)  = min 

(n-1)  -:t<NC 


j".*  E 


+ (n-1 ) Cg+F+oG 


(5.66) 


where  A and  B are  constant  (in  the  region  where  v+2  is  the 
trapping  state)  with  values 
NC 

A = E <5'67 

m=n+l 

NC  NC 

B = (n-1)  Y]  9 + Y 0m  (5.68 

m=l  m=n+l 


and  F and  G are  also  constant  (in  the  region  where  v is 


Now  we  have  all  the  necessary  elements  to  study  the 


exact  shape  of  the  curves  of  total  cost  versus  storage  cost 
taking  the  updating  rate  as  a parameter  or  otherwise  total 


cost  versus  updating  ratio  taking  the  storage  cost  as  a 


For  p=const.V*  (1)  is  a picewise  continuous  linear 


that  will  change  its  slope  every  time  we 


function  of 


a new  minimum  in 


will  happen  every  time  C 9 for  n+l<m  £ NC) 


const 


constant  for  a fixed  a 


is  constant  in  the  zone  between  equilibrium  lines  and 


thererore  V*(l)  is  a picewise  continuous  linear  function 


of  p that  will  only  change  its  slope  every  time  it  crosses 
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With  these  properties  we  can  now  draw  the  exact  cost 
curves  for  the  example  of  this  section,  without  implement- 
ing the  dynamic  program  algorithm.  The  curves  are  in  Fig. 

V . 8 a)  and  b) . This  figure  closes  the  analysis  of  networks 
with  constant  parameters  and  updating  traffic  and  in  the 
next  section  the  analysis  of  the  computer  networks  with 
constant  parameters,  updating  traffic  and  some  nonzero 
probability  of  node  failures  will  be  considered. 

V. 3 . Nonzero  Failure  Probability 

The  dynamic  analysis  under  consideration  takes  its 
complete  meaning  when  we  include  in  the  system  the  probability 
of  node  failure. 

The  Markov  process  describing  the  evolution  of  the  system 
without  failures  under  optimal  decision  rules  has  a trapping 
state,  a fact  which  gives  rise  to  a number  of  specific 
properties  as  described  in  section  V.2.  On  the  other  hand, 
if  failures  occur,  the  trapping  state  disappears,  as  seen 
from  the  discussion  in  Chapter  IV.  In  particular,  if 
failures  can  happen  in  all  computers  with  nonzero  probability, 
then  the  steady  state  probability  that  the  Markov  process 
will  be  in  a given  state  is  strictly  less  than  1 for  any 
decision  strategy. 

Let  us  analyze  a simple  example.  Consider  a network 
with  two  computers,  NC=2,  and  the  following  parameters: 


8a.  Curves  of  total  minimum  cost  vs.  storage  cost  with  p as  a parameter 
i represents  trapping  state  and  j optimum  initial  state. 


Curves  of  the  total  minimum  cost  vs.  p with  C as  a parameter 
i represents  trapping  state  and  j optimum  initial  state. 
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where  under  "cost"  we  write  the  values  of  the  costs-to- 
go  ordered  from  state  0 in  top  to  state  8 in  bottom, 
after  every  iteration.  Under  "go  to"  we  list  the  opti- 
mal decision,  corresponding  to  every  state.  The  stars 
mean  that  for  these  states  no  decisions  are  needed. 

Continuing  the  iterations  we  could  see  that  the 
decision  "go  to  state  4"  is  in  fact  the  steady  state 
optimal  solution  for  states  1,  3 and  4.  This  fact  can 
also  be  confirmed  by  applying  the  Howard  algorithm  20 
to  this  problem. 

Therefore,  for  this  problem,  with  the  specified 
parameters,  the  optimal  steady  state  decision  is  to  keep 
as  many  copies  as  possible,  that  is  one  copy  at  each 
computer . 

If  keeping  now  the  other  parameters  fixed  we  reduce 
the  probability  of  failure  by  a factor  of  10,  i.e.  from 
0.01  to  0.001,  we  will  find  that  the  optimal  steady 
state  decision  is  "go  to  state  3",  that  is  keeping  only 
one  copy  at  computer  one.  The  first  iterations  are 
shown  in  table  V.6. 
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Therefore  a reduction  in  the  risk  of  node  failures 

! 

is  reflected  in  a reduction  in  the  number  of  copies.  It 
is  obvious  that  further  reductions  in  the  probability  of 
failures  will  not  have  any  influence  at  all  in  the  optimum 
policy  because  we  cannot  have  less  than  one  copy  in  the  system. 

We  have  seen  how  changes  in  the  failure  probability 
may  change  the  minimum  number  of  copies  to  be  kept  in  a 
system;  but  what  will  be  the  repercussion  of  changes  in  Pr, 
the  probability  of  recovery?  It  can  be  easily  seen  that 
changes  in  P^.  will  not  have  as  much  influence  in  the  optimal 
decision  as  changes  in  P^  for  the  case  NC=2.  The  reason  is 
that,  for  NC=2,  P^  only  appears  in  transition  probabilities 
from  states  where  no  decisions  are  available.  The  influence 
of  these  state  costs  on  the  decisions  from  other  states  is 
reflected  through  smaller  probabilities  (assuming  Pf  relative- 
ly small)  and  in  a relatively  simmetric  form.  Only  if  Pf 
is  near  1,  the  value  P might  have  a certain  importance. 

An  intuitive  explanation  of  the  fact  that  the  value  of 
Pj  will  have  much  more  influence  than  the  value  of  Pr  can 
be  given  by  observing  that,  no  matter  how  fast  the  failed 
computers  have  been  restored,  if  the  system  looses  all  the 
copies,  then  a high  price  has  to  be  paid  in  order  to  bring  a 
copy  from  outside. 

For  cases  with  NC>2  the  situation  is  not  so  simple 
because  then  P may  appear  directly  in  transition  probabi- 
lities with  several  decisions  available.  Nevertheless,  we 
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can  say  that  in  those  cases,  the  larger  the  number  of  work- 
ing computers  in  a particular  state  the  smaller  the  influence 

of  P , for  P_<<1. 
r f 

Therefore  Pr  will  affect  more  the  decisions  among  states 
with  large  number  of  failed  computers.  The  reason  can  be 
easily  seen  with  one  example.  Consider  a network  with  NC=5 


computers.  If  we  are  in  state  1 


(00001)  then  the  next 


important  costs  that  will  affect  the  optimum  decision  from 
1 will  be  the  costs  of  states  with  no  failures  in  their 
components,  and  therefore  no  Pf  in  their  probability  expres- 
sions; all  these  costs  are  going  to  be  multiplied  by  (1-P^)3, 
(remember  P^<<1).  The  next  set  of  important  states  contribut- 
ing to  the  decisions  is  the  set  of  states  with  one  failed 

4 

computer.  These  terms  will  have  a factor  of  (1-Pj)  P^. 

Continuing  in  this  way  we  can  see  that  the  states  that  will 

reflect  more  the  value  of  P^,  that  is  the  states  with  a large 

number  of  2's  in  their  components,  will  be  multiplied  by 

very  small  weights;  for  instance  a state  with  only  one  working 

computer  will  be  affected  by  the  term  P^^(l-P^)-C  if  P^<<1- 

On  the  other  hand  if  the  present  state  is  for  instance, 

state  (0  2 1 2 2),  and  we  assume  P >>Pf.  then  the  important 

2 3 

terms  will  be  the  terms  affected  by  the  weights  (1-P^)  (1-Pr)  , 

(1-Pf  (2Pr  (1-Pr ) 2 and  (l-P^JP^  (1-P^)  3 and  therefore  P^.  will 


f'* f ' r 


increase  its  role  in  the  optimal  decision  from  these  states 
compared  to  the  former  one. 

An  intuitive  explanation  to  this  fact  can  be  given  as 


fol lows : 


, . 
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If  Pr  has  a high  value,  close  to  1,  then  there  are  high 
probabilities  of  transitions  from,  say  state  (02122)  to 
states  like  ( 0 0 0)  or  (_  2 _ 0 0) ; suppose  now  that  com- 

puters 4 and  5 have  high  request  rates  and  that  transmis- 
sion costs  from  computer  3 to  computer  4 and  5 are  much 
higher  than  transmission  costs  from  computer  1 to  the  same 
computers,  then  a decision  of  writing  a copy  at  computer  1 
will  probably  be  optimal.  On  the  other  hand,  if  is  close 
to  0 then  the  transition  probabilities  to  the  statesabove 
will  be  very  small  and  other  factors  will  influence  the  optimal 
decision . 

Of  course  if  P^  is  near  1 then  Pr  will  increase  its 
role  in  all  decisions.  With  the  discussion  above  we  have 
only  confirmed  that  the  model  in  fact  reflects  the  physical 
intuition  that  as  long  as  P^  remains  very  small,  the  proba- 
bility of  recovery  is  of  no  great  importance  in  the  system 
(remember  the  intuitive  explanations  given  above) . It  is 
obvious  that  the  previous  discussion  has  been  undertaken 
considering  a fixed,  not  too  small,  cost (comparing  to  trans- 
mission costs)  of  bringing  a copy  from  outside  to  the  system, 
in  the  case  of  loosing  all  the  copies.  It  is  clear  that 
these  costs  will  play  a similar  but  opposed  role  to  the 
failure  probabilities , The  reason  is  trivial. 

Let  us  consider  now  an  example  with  3 computers  NC=3. 

The  parameters  of  the  network  are  the  following: 


0.8 


02(t)  = 02  = 0,6 
e3(t)  = ©3  = 0.4 

C.  . = CT  = 1 Vi, j i, j {1,2,3} 

1 > J 

03=02=03=03=  0.25 
P = 0.25 

cQ1  = 1000  cQ2  = 1001  cQ3  = 1002 

Pr  = 0.1 

Now  we  write  the  optimal  decisions  corresponding  to 
the  first  iteration  for  three  values  of  P^. 

Pf  = 0.01,  0.001,  0.0001 

In  all  cases  it  can  be  checked  that  the  decisions  at 
time  T-5  (and  T-4)  constitute  already  the  steady  state  optimal 
policy . 

In  these  three  cases  it  can  be  very  well  seen  how  the 
decrease  of  the  failure  probability  decreases  the  number  of 
copies . 

For  P^=0.01  the  steady  state  policy  is  to  keep  always 
as  many  copies  as  possible,  that  is,  all  working  computers 
will  carry  a copy.  For  Pf  = 0.001  the  optimal  policy  is  to 
keep  as  many  copies  as  possible  except  for  state  12  and  13 
where  two  copies  are  enough  (remember  the  discussion  on  Table 
V. 3) . For  P^  = 0.0001  the  optimal  policy  says:  if  all  com- 
puters are  working  keep  only  two  copies  (in  1 and  2) , other- 
wise keep  as  many  copies  as  possible. 

The  optimal  decision  from  states  12  and  13  for  the  case 


Pf=0.001  might  appear  surprising  considering  that  from  all 
other  states  the  decision  is  to  go  to  state  13,  The  reason 
for  this  apparently  anomalous  fact  can  be  seen  by  observing 
that  a direct  movement  to  state  12  carries  a greater  risk  of 
ending  up  in  a state  with  only  one  copy,  than  if  we  try  to 
move  the  system  to  state  13.  Then  the  transition  from  13  to 
12  is  automatic  (see  also  the  discussion  on  Table  V.3). 

Another  fact  that  can  be  inferred  from  the  table  is  that 
the  optimal  decisions  always  try  to  keep  copies  in  the  com- 
puters with  highest  request  rate  and  smallest  cost  of  bringing 
a copy  from  outside.  Remember  that  in  the  present  example 

We3  and  C01<C02<C03 

Looking  at  all  the  examples  studied  in  this  section  we 
can  see  as  a common  point  that  in  all  cases  the  steady  state 
optimal  solution  is  reached  after  very  few  iterations.  If  we 
began  the  iterations  with  a set  of  different  terminal  costs 
this  would  not  have  been  the  case.  This  fact  has  been  con- 
firmed for  several  examples  by  evaluating  eigenvalues  and 
eigenvectors  of  the  transission  matrix.  We  expanded  dif- 
ferent terminal  cost  vectors  in  the  matrix  eigenvector  base 
and  observed  that  the  vector  A gives  the  quickest  rate  of 
convergence.  Nonetheless,  this  fact  could  not  be  proven 
analytically.  A good  reason  to  believe  that  the  chosen 
terminal  costs  are  a good  set  of  values  fcr  a good  speed  of 
convergence  of  the  iterations  is  to  think  that  with  these 
terminal  costs  we  let  the  system  finish  ir.  a"natural  and 
not  forced"  way  because  those  terminal  costs 


are  similat  to 


rj 
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the  immediate  cost  (except  for  certain  corrections  due  to 
the  CQi  costs  that  make,  in  general,  the  first  decision 
in  the  iterations  different  from  the  others).  Nevertheless, 
this  fact  will  tell  us  that  those  terminal  vectors  are 
better  than  others  but  not  necessarily  giving  rise  to  a 
quick  convergence. 

If  this  is  indeed  the  case,  we  could  say  that  because 
we  are  mainly  interested  in  the  steady  state  policy  the  elec- 
tion of  the  per  unit  time  costs  as  terminal  costs  will  reduce 
considerably  the  amount  of  computation  needed  to  find  the 
steady  state  optimal  policy.  Otherwise,  we  always  have 
available  the  algorithm  developed  by  Howard  £20]  that  has 
been  proven  quite  efficient  for  those  kinds  of  problems. 

In  order  to  implement  this  algorithm  we  could  take  advantage 
of  the  fixed  zeroes  position  in  the  transition  matrix  to 
solve  the  system  of  equations  that  this  system  generates. 
Another  point  that  can  save  certain  amount  of  computation  in 
the  solution  of  the  system  of  equations  is  the  fact  that 
all  states  with  NC=1  "2"'s  in  the  equivalent  positions  give 
rise  to  identical  transition  probabilities  and  hence  to 
identical  rows  in  the  transition  matrix.  That  is  for  NC=4 . 


79= (2221) 
78= (2220) 


row  79  of  P(t,u)  = row  78  of  P(t,u)  Vt,u 


77= (2212) 
74= (2202) 


row  77  of  P(t,u)  = row  74  of  P(t,u)  Vt,u 


etc . 


There  have  also  been  suggested  in  the  literature  e.g.^23] 
linear  programming  formulations  to  obtain  an  optimal  policy 
using  the  principles  of  the  policy  improvement  procedure  of 
Howard's  algorithm;  these  formulations  solve  the  maximization 


problem  involved  in  the  policy  improvement  iteration  of 
Howard's  algorithm  by  means  of  linear  programming  calculations 


V.4  Completely  Simmetric  Network 


In  spite  of  the  simplicity  that  the  model  provides  to 
the  analysis  of  a general  network  with  any  number  of  compu- 
ters, there  is  one  argument  against  it,  as  in  fact  is  the 
case  with  many  finite  state  formulations.  The  problem  is 
the  exponential  growth  of  the  number  of  states  with  the  number 


of  computers  in  the  network  grows.  There  is  not  much  that 


can  be  done  in  order  to  avoid  this  growth  but  to  try  to  find 


suboptimal  solutions.  One  way  in  which  these  suboptimal 


schemes  can  be  found  is  to  assume  that  all  transmission  costs 


are  equal,  all  rates  are  equal,  all  storage  costs  are  equal 


etc.  In  such  a situation  we  can  see  that  all  nodes  in  the 


network  have  the  same  role  and  there  is  no  need  to  specify 


which  node  or  nodes  have  a copy  at  a certain  time  but  instead 


the  state  vector  will  contain 


number  of  copies  in  the  system  at  any  particular  time.  Then 


the  general  Markov  process,  for  this  network,  is  mergeable 


in  the  sense  that  we  can  g 


number  of  states  into  a superstate  and  work  with  this  super 
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state  as  if  it  was  a simple  state  for  the  modified  Markov 
process.  In  the  case  that  for  certain  reasons,  important 
difference  in  computer  request  rates  for  instance,  we  were 
interested  in  a further  analysis,  within  some  superstates, 
this  can  be  done  after  the  merged  process  was  analyzed.  Of 
course  the  results  are  not  going  to  be  optimal  any  longer 
but  if  we  choose  wisely  the  parameters  good  bounds  to  the 
optimal  situation  could  be  obtained. 

For  the  case  NC=2  we  know  that  the  states  are  0-8, 
and  can  be  grouped  or  merged  as  it  is  shown  in  (5.72) 


0-0  0 0 


It  can  be  seen  that  this  grouping  verifies  all  the  pro- 
perties needed  for  a right  merging,  that  is 


£ Pi,m  " £ Pjfjn 

mes^  mes^ 


for  i, j e Sk  (5.73) 

where  S^,  SjS  {0,A,B,C,D} 


because  now  the  indeces  in  the  control  variables  are  meaning- 
less and  ©^  = ©2  = © 

The  elements  of  the  new  transition  matrix  will  be 


u >»- 
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0 A 


B 


C 


D 


( 1-Pf ) ^ 
(1-Pf)2 
(1-Pf)2 
(l-Pf)P 
0 


(1-a©) 

c 

rj 


0 


(1-Pf) ‘a© 

( 1-Pf ) 2 ( 1-e ) 
0 
0 


2Pf (1-Pf) 

2Pf ( 1 — P f ) 

2Pf(1_pf} 

(1-Pf)  (l-Pr)^PfPr 

2Pr (1-Pr) 


Pf(3-Pr) 

(1-Pr)2 


where  now  a represents  the  fact  of  adding  one  copy  and  e the 
fact  of  erasing  a copy,  no  matter  in  which  computer  these 
actions  will  take  place. 

As  an  example  let  us  see  that  if  fact  P^  = ( 1-P^ ) 2 ( l-a©) • 
We  can  write  from  the  properties  of  the  merging  process 


P =p  +p  =p  +p 

*AA  l , li , 3 *3,1  *3,3 


(5.75) 


where 


1,1 

1.3 
3, 

3.3 


(1-Pf) 2 (1-aj©) 
(1-Pf) 2c2ai© 
d-Pf)  2c1a2© 

(1-Pf) 2(l-a2©) 


(5.76) 


omitting  the  indeces,  that  we  saw  are  now  meaningless, 

P = (1-Pf ) 2 (1-ae+ra©)  (5.77) 

r\  f A 1 

but  because  the  decision  of  going  to  state  0 is  not  an  admis 
sible  decision  and,  furthermore,  we  do  not  want  to  allow 
contradictory  decisions  (erasing  and  writing  simultaneously) 
we  have 

e = 0 


.74 
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1 


and  then 


PA,A  = (1-Pf>  (1-a8) 


(5.78) 


We  can  see  that  with  this  rearrangement  we  have  reduced 
a 9 state  system  to  a 5 state  system. 

For  NC=3  the  reduction  is  even  more  drastic.  The  states 
can  be  grouped  as  follows 


New  state 


Grouped  state 


Representative  of 
the  new  state 


0 

0 

(0 

0 

0) 

A 

1,3,9 

(1 

0 

0) 

B 

4,10,12 

(1 

1 

0) 

C 

13 

(1 

1 

1) 

D 

2,5,6,7,11,15,18,19,21 

(0 

0 

2 ) and ( 1 

E 

14,16,22 

(1 

1 

2) 

F 

8,17,20,23,24,25 

(0 

2 

2 ) and ( 1 

G 

26 

(2 

2 

2) 

(5.79) 


So  we  had  a reduction  from  27  to  8 in  the  number  of 
states.  Now  we  had  to  define  the  new  control  variables, 
add  a new  copy 

2 


a 


add  two  new  copies 


(5.80) 


r.  erase  one  copy 

2 

c erase  two  copies 

The  process  can  be  easily  generalized. 
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CONCLUSIONS  AND  OPEN  QUESTIONS 


Throughout  the  chapters  of  this  thesis  we  have  developed 
a new  model  to  handle  the  problem  of  optimal  dynamic  file 
allocation.  The  model  had  to  be  general  enough  to  allow  the 
study  of  problems  such  as:  dynamic  allocation  with  possible 
computer  failure  and  optimal  allocation  when  we  have  res- 
trictions in  the  state  space.  The  restrictions  may  take 
the  form  of  a maximum  number  of  copies  allowed  in  the  system 
at  any  instant  of  time  or  not  allowing  copies  of  the  file 
simultaneously  in  two  or  more  given  computers.  The  use  of 
two  types  of  control  variables,  one  for  adding  new  copies 
to  the  system  (a),  and  the  other  for  erasing  copies  (c), 
made  easier  the  task. 

First  we  stated  the  working  hypothesis  (suf f icientlv 
high  link  capacities,  sufficient  memory  sizes,  stochastic 
independence  in  the  requesting  process  from  different  com- 
puters, etc.)  that  could  allow  us  to  work  with  each  file 
separately  and  to  model  the  system  as  a Markov  process. 

Having  characterized  the  evolution  of  the  system  under 
a Markov  process  and  being  interested  in  finding  the  optimal 
dynamic  file  allocation  such  that  the  total  cost  were  minimized 
we  found  in  the  stochastic  dynamic  programming  an  excellent 
tool  to  solve  the  problem. 

We  defined  the  state  of  the  system  as  a vector  with 
a number  of  components  equal  to  the  number  of  computers  in 
the  network.  In  that  way  each  component  of  the  vector  would 
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characterize  the  particular  situation  of  each  computer.  We 
would  have  a 0 in  the  ith  location  if  computer  i were  in 
working  condition  but  with  no  copy  stored  in  its  memory; 
a 1 if  computer  i had  a copy  in  its  memory  and  a 2 if  computer 
i were  out  of  work  (if  failures  are  included  in  the  model) . 
With  this  representation  we  could  think  in  the  state  of  the 
system  as  being  a base  3 reproduction  of  a certain  decimal 
number  (or  base  2 reproduction  when  the  probability  of  failure 
is  considered  equal  to  zero) . Therefore  we  could  identify 
the  states  with  this  number. 

We  showed  that  the  states  and  control  vectors  exhibited 
some  properties  tl it  allowed  us  to  write  mechanically  the 
transition  tableau.  This  transition  tableau  has  proved  of 
great  utility  in  writing  the  recursive  equations  generated 
by  the  application  of  dynamic  programming.  In  fact  we 
found  some  rules  that  made  it  possible  to  construct  algorith- 
mic flow-charts  to  compute  the  transition  probabilities  in 
a very  efficient  way.  Perhaps  one  of  the  next  important 
points  related  with  this  algorithm  is  its  property  of  being 
totally  general  as  far  as  the  number  of  computers  is  concerned. 

We  have  also  seen  that  updating  traffic  generated  at 
some  or  all  of  the  nodes  can  be  easily  incorporated  in  the 
analysis.  We  have  given  flow-charts  showing  how  all  these 
terms  can  be  calculated  in  the  same  way  as  the  per  unit  time 
costs . 

One  of  the  reasons  why  the  flow-charts  were  found  to  be 
quite  efficient  is  because  they  compute  only  the  nonzero 
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elements  of  every  matrix  and  vector . This  is  an  important 
fact  if  we  consider  that,  for  instance,  in  the  transition 
matrix  only  approximately  30%  of  the  components  are  nonzero. 

A flow-chart  for  the  whole  optimization  process  was  also 
presented . 

After  the  complete  introduction  of  the  model  we  applied 
it  to  several  examples.  First  we  consider  the  case  of  time 
varying  rates  with  no  failures  and  no  updating  traffic  in 
the  network.  We  studied  for  this  case  how  the  state  dyna- 
mics changed  as  the  storage  cost  was  increased.  The  analysis 
confirmed  the  intuitive  point  that  the  maximum  number  of 
copies  needed  in  the  network  decreased  as  the  storage  cost 
increased.  We  also  compared,  for  this  case,  the  dynamic 
analysis  with  some  static  analysis.  We  plotted  the  curves 
of  total  cost-versus  storage  cost  for  the  static  and  dynamic 
analyses  for  two  examples  with  different  rates  but  with  the 
same  average  rates  over  the  period  of  operation.  It  was 
found  that  the  curves  for  the  dynamic  analysis  were  very 
close.  A third  example  with  higher  rates  were  also  plotted 
showing  higher  costs. 

Later  on,  the  case  of  constant  rates  with  updating  traf- 
fic and  no  failure  was  studied  in  great  detail.  It  was 
shown  that  for  this  case  the  ” irkov  process  fitted  into  the 
special  class  of  Markov  chains  with  a trapping  state.  This 
fact  was  used  to  derive  a certain  number  of  properties.  One 
of  the  properties  is  that  these  processes  with  a terminal 
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cost  vector  equal  to  the  immediate  cost  vector  do  not  present 
any  transient  in  their  decision  policy.  Furthermore,  the 
minimum  expected  cost  increases  picewise  linearly  with  time 
to  go.  One  important  outcome  of  these  two  results  was  to 
find  some  expressions  relating  the  trapping  state  and  the 
optimal  initial  state  with  the  storage  cost  and  the  updating 
ratio.  In  that  way  we  could  study,  without  actually  implement- 
ing the  dynamic  programming  algorithm,  how  the  optimal 
allocation  changed  as  we  vary  either  the  storage  cost  or 
the  updating  ratio  or  both.  Curves  were  also  given  to  show 
the  evolution  of  the  total  cost  versus  storage  cost  taking 
the  updating  ratio  as  a parameter,  and  the  total  cost  versus 
updating  ratio  taking  the  storage  cost  as  a parameter.  It 
was  shown  that  due  to  above  mentioned  properties,  those 
curves  could  be  drawn  without  actually  using  the  dynamic 
programming  algorithm. 

Finally  the  case  of  nonzero  failure  probability  was  studied. 
It  was  shown  how  an  increment  in  the  failure  probability  may 
increase  the  number  of  copies  to  be  stored  in  the  network  at 
any  time.  It  was  also  shown  how,  in  general,  variations  in 
the  probability  of  computer  recovery  do  not  have  a signifi- 
cant effect  on  the  optimal  decisions  if  the  failure  probability 
is  reasonably  small.  Perhaps  one  of  the  most  important  find- 
ings for  this  case,  that  could  not  be  proved  analytically 
though,  was  the  fact  that  taking  the  per  unit  time  cos>.  as 
the  vector  terminal  costs  the  process  converges  very  quickly 
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to  the  steady  state  decision.  This  is  an  important  fact 
that  could  save  a lot  of  computation  and  would  avoic  the 
need  to  use  the  Howard  algorithm.  Otherwise  Howard's  algorithm 
could  be  used  efficiently  while  taking  advantage  of  the  fixed 
zero  position  in  the  transition  matrix  and  the  fact  that  some 
of  its  rows  are  identical. 

As  it  was  pointed  out  earlier,  one  of  the  difficulties 
that  the  finite  state  model  rises  is  the  fast  increase  in 
the  number  of  states  when  the  number  of  computers  increases. 

To  overtake  this  difficulty  a suboptimal  method  based  on  a 
completely  simmetric  network  which  can  be  thought  as  an 
approximation  to  the  actual  network  was  suggested.  This 
approximation  provides  a reduced  Markov  chain  whose  state 
are  collections  of  the  states  of  the  original  process. 

The  reduction  in  the  number  of  states  for  NC=3  was  from  27  to  8. 

Some  points  remain  still  to  be  studied  related  to  dynamic 
file  allocation.  As  mentioned  before,  we  found  some  con- 
vergence properties  that  could  not  be  proved  analytically. 
Furthermore  other  suboptimal  models  can  be  of  interest  for 
the  case  of  large  networks.  For  example,  some  a priori 
calculated  bound  in  the  maximum  and  minimum  number  of  copies 
could  reduce  considerably  the  number  of  states.  But  per- 
haps one  of  the  most  appealing  topics  to  be  pursued  in  this 
area  is  including  the  situation  when  the  rates  of  request  are 
not  perfectly  known  in  advance.  The  main  goal  then  would 
he  to  try  to  generalize  Segall's  results  11  for  this  problem 


to  a broader  framework  as  the  one  presented  here  for  the  case 
of  deterministic  rates.  With  this  approach,  one  could  also 
investigate  decentralized  schemes  for  dynamic  file  allocation 
where  the  decisions  at  every  time,  whether  to  write  or 
erase  a copy,  are  done  locally  by  each  computer  and  all 
computers  work  in  a team  to  minimize  the  overall  cost  [28] . 
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APPENDIX  A 


Transition  Probabilities  for  NC=2 


We  defined 

P£j (t,u)  = ProbfY (++1)  = j Y ( t ) = i) 

In  particular 

Plx(t,u)  * Prob(Y (t+1)  = 1 Y ( t ) = 1}  = 

= Prob{ Y (t+1 ) =1  Y (t)=l,n1 (t)=l}Prob{n1 (t)=l  Y(t)=l} 
+ Prob{Y (t+1) =1  Y(t)=l,n1 (t)=0)Probfn,  (t)=0  Y(t)=l} 

but 

Prob{n1 (t ) =1 |Y(t)=l}=  Prob{n1 (t)=l}=  ©1(t) 

Prob{n1(t)=0|Y(t)  = l}=  i-e^t) 

Prob{Y(t+l)=l|Y(t)-l,n1  (t)=l}=(l-a1  (t))(l-e2  (t)  ) 

Prob{Y(t+l)=l |Y(t)=l, n1 (t)=0}=  1 

therefore 

Pn  (t/u)  = (l-a1(t)  ) (l-«r2(t))  01(t)  + 1-©1  (t)  = 

= (l-al(t))©1  ( t)  - ( l-ct1  (t)  )c2(t)©1  (t)  + l-©1  (t) 
but  (1-a^ (t)  ) e2 (t)  - 0 for  any  value  of  a^(t) 
and  e2(t)  i-n  t^le  contr°l  space  then 
P1X ( t , u ) = 1 -«1 (t)©1 (t) 

In  the  same  way 

P12 (t , u)  =Prob( Y (t+ 1) =2 | Y (t) =1  } = 

=Prob{ Y ( t+1 ) =2 | Y ( t ) =1, n1 ( t ) = 1 }Prob{ n ( t ) = 1 }+ 
+Prob{Y(t+l)=2 | Y ( t ) =1 , n ^ ( t ) =0 } Prob{ n ^ ( t ) =0 } 
ProbfY(t+l)=2  |Y(t)=l,n  ( t ) =0 >=  0 


MB -fli' 


but 
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APPENDIX  B 

Backward  Equations  for  a Network  with  Two  Computers  and  the 
Restriction  of  Only  One  Copy  in  the  System. 

We  saw  in  section  III-3  that  the  backward  equations 
for  the  case  NC=2 without  restrictions  in  the  state  space 
were 


V*  (t)" 

C2+C210l (t) 

v*(t) 

- 

Cl+C1202(t) 

V*  (t) 

c1+c2 

1-a*  (t)©1 (t)  e*(t)a*(t)©1(t)  (1-e* (t) ) a* (t)©1 (t) 

F.*(t)a*(t)G2(t)  1-a*  (t)02  (t)  (l-c*(t))a*(t)02(t) 

e*(t)  e*(t)  l-e*(t)-e*(t) 

If  we  restrict  the  system  to  have  only  one  copy  at 
any  instant  of  time  then  state  3 is  not  allowed  and  we  have 
to  do  two  changes  according  to  section  III-5. 

1)  we  eliminate  the  last  row  of  every  matrix  in  the 
equation 

2)  we  add  the  probability  of  going  to  state  3 to  the 
probability  of  remait.ing  in  the  previous  state. 

Doing  that  the  transition  state  becomes 


V* (t+1) 

v* (t+1 ) 
v* (t+i: 


Realizing  now  (looking  at  the  tableau  of  section  1 1 — 4 ) 


that  for  any  allowed  transition  a 


not  need  the  erasure  variables.  With  this  simplification 


the  recursive  equation  is 


that  is  the  same  equation  of  ref.  11  except  for  the  switching 


of  subscripts  due  to  different  state  definitions 


V*  (t) 

II 

V*  (t) 

l 
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